2024-03-08T23:44:49.493 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-08T23:44:49.493 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --archive /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} --owner scheduled_rishabh@teuthology --name rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi -- /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/orig.config.yaml 2024-03-08T23:44:49.551 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-08T23:44:49.633 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328 branch: rishabh-2024mar8 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} email: null first_in_suite: false job_id: '7588328' kernel: client: sha1: distro flavor: default kdb: true sha1: dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412 ktype: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: rishabh-2024mar8 ceph: cephfs: ec_profile: - m=2 - k=2 - crush-failure-domain=osd max_mds: 1 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug cephsqlite: 20 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 7d63482aa89628d733e298984a762389b625c761 subvols: create: 2 subvol_options: --namespace-isolated --size 25000000000 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.root_rsnaps - mds_server.req_mksnap_latency.avgcount - mds_server.req_rmsnap_latency.avgcount - mds.dir_split install: ceph: flavor: default sha1: 7d63482aa89628d733e298984a762389b625c761 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=legacy - nowsync syntax: v1 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: rishabh-2024mar8 sha1: 7d63482aa89628d733e298984a762389b625c761 owner: scheduled_rishabh@teuthology priority: 70 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 1249 sha1: 7d63482aa89628d733e298984a762389b625c761 sleep_before_teardown: 0 subset: 481/512 suite: fs suite_branch: rishabh-2024mar8 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 7d63482aa89628d733e298984a762389b625c761 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPscUV6Gm6IOLkVUrW7zI6I49cIDeUJYfmsfXdJ+pGNir7An9u17Nx42xmBpog0CGG3wCojiytGRM3ZJfqHE+A= smithi102.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2GvsZBXQDLL/W3R/OhFX9IjxjwLYTlsB037IJvqxbqbKWiyFE+B4hCNfXcx1TRCjGJk3Gy+B1paKxL2hqjlF0= smithi179.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9LxVk+vl16XT+ugVB8cplHsXPPuF9kA+ghz4vJ3M+6u8igdtPgHDPWuSv8zrS7ViQdzMzMzIcf0UTDE/QEw0o= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - exec: mon.a: - ceph mgr module enable snap_schedule - ceph config set mgr mgr/snap_schedule/allow_m_granularity true - ceph config set mgr mgr/snap_schedule/dump_on_update true - ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m - ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - date +%s > START_TIME - full_sequential_finally: - exec: mon.a: - date +%s > END_TIME - START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi - ceph fs snap-schedule status --fs=cephfs --path=/ - ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/fsx.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-08_19:54:47 tube: smithi user: rishabh verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-08T23:44:49.633 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa; will attempt to use it 2024-03-08T23:44:49.634 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks 2024-03-08T23:44:49.634 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-08T23:44:49.635 INFO:teuthology.task.internal:Checking packages... 2024-03-08T23:44:49.659 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '7d63482aa89628d733e298984a762389b625c761' 2024-03-08T23:44:49.659 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-08T23:44:49.659 INFO:teuthology.packaging:ref: None 2024-03-08T23:44:49.659 INFO:teuthology.packaging:tag: None 2024-03-08T23:44:49.659 INFO:teuthology.packaging:branch: rishabh-2024mar8 2024-03-08T23:44:49.659 INFO:teuthology.packaging:sha1: 7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:44:49.659 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=rishabh-2024mar8 2024-03-08T23:44:49.871 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1965.g7d63482a 2024-03-08T23:44:49.872 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-08T23:44:49.879 INFO:teuthology.task.internal:no buildpackages task found 2024-03-08T23:44:49.879 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-08T23:44:49.886 INFO:teuthology.task.internal:Saving configuration 2024-03-08T23:44:49.913 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-08T23:44:49.919 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-08T23:44:49.945 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.163132', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPscUV6Gm6IOLkVUrW7zI6I49cIDeUJYfmsfXdJ+pGNir7An9u17Nx42xmBpog0CGG3wCojiytGRM3ZJfqHE+A='} 2024-03-08T23:44:49.969 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.161609', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2GvsZBXQDLL/W3R/OhFX9IjxjwLYTlsB037IJvqxbqbKWiyFE+B4hCNfXcx1TRCjGJk3Gy+B1paKxL2hqjlF0='} 2024-03-08T23:44:49.993 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.164399', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9LxVk+vl16XT+ugVB8cplHsXPPuF9kA+ghz4vJ3M+6u8igdtPgHDPWuSv8zrS7ViQdzMzMzIcf0UTDE/QEw0o='} 2024-03-08T23:44:49.993 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-08T23:44:50.000 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-08T23:44:50.000 INFO:teuthology.task.internal:roles: ubuntu@smithi102.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-08T23:44:50.000 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-08T23:44:50.000 INFO:teuthology.run_tasks:Running task console_log... 2024-03-08T23:44:50.102 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa4bc2fa0d0>, signals=[15]) 2024-03-08T23:44:50.103 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-08T23:44:50.113 INFO:teuthology.task.internal:Opening connections... 2024-03-08T23:44:50.113 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-03-08T23:44:50.117 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:44:50.247 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi102.front.sepia.ceph.com 2024-03-08T23:44:50.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:44:50.342 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2024-03-08T23:44:50.343 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:44:50.438 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-08T23:44:50.444 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-03-08T23:44:50.462 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-03-08T23:44:50.462 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-03-08T23:44:50.519 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="CentOS Stream" 2024-03-08T23:44:50.519 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:ID="centos" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE="rhel fedora" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:PLATFORM_ID="platform:el9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:ANSI_COLOR="0;31" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:LOGO="fedora-logo-icon" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://centos.org/" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T23:44:50.520 INFO:teuthology.orchestra.run.smithi073.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T23:44:50.521 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-03-08T23:44:50.542 DEBUG:teuthology.orchestra.run.smithi102:> uname -m 2024-03-08T23:44:50.559 INFO:teuthology.orchestra.run.smithi102.stdout:x86_64 2024-03-08T23:44:50.559 DEBUG:teuthology.orchestra.run.smithi102:> cat /etc/os-release 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:NAME="CentOS Stream" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION="9" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:ID="centos" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:ID_LIKE="rhel fedora" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:VERSION_ID="9" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:PLATFORM_ID="platform:el9" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:ANSI_COLOR="0;31" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:LOGO="fedora-logo-icon" 2024-03-08T23:44:50.615 INFO:teuthology.orchestra.run.smithi102.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T23:44:50.616 INFO:teuthology.orchestra.run.smithi102.stdout:HOME_URL="https://centos.org/" 2024-03-08T23:44:50.616 INFO:teuthology.orchestra.run.smithi102.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T23:44:50.616 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T23:44:50.616 INFO:teuthology.orchestra.run.smithi102.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T23:44:50.616 INFO:teuthology.lock.ops:Updating smithi102.front.sepia.ceph.com on lock server 2024-03-08T23:44:50.642 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2024-03-08T23:44:50.660 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2024-03-08T23:44:50.661 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="CentOS Stream" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="9" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:ID="centos" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE="rhel fedora" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="9" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:PLATFORM_ID="platform:el9" 2024-03-08T23:44:50.717 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:ANSI_COLOR="0;31" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:LOGO="fedora-logo-icon" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://centos.org/" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-08T23:44:50.718 INFO:teuthology.orchestra.run.smithi179.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-08T23:44:50.718 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2024-03-08T23:44:50.742 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-08T23:44:50.753 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-08T23:44:50.783 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-08T23:44:50.784 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-03-08T23:44:50.787 DEBUG:teuthology.orchestra.run.smithi102:> test '!' -e /home/ubuntu/cephtest 2024-03-08T23:44:50.789 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2024-03-08T23:44:50.805 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-08T23:44:50.811 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-08T23:44:50.811 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-03-08T23:44:50.843 DEBUG:teuthology.orchestra.run.smithi102:> test -z $(ls -A /var/lib/ceph) 2024-03-08T23:44:50.845 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2024-03-08T23:44:50.956 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-08T23:44:51.033 INFO:teuthology.run_tasks:Running task kernel... 2024-03-08T23:44:51.049 INFO:teuthology.task.kernel:normalize config orig: {'client': {'sha1': 'distro'}, 'flavor': 'default', 'kdb': True, 'sha1': 'dd5c1cd171679b4e0ceeab3e8d62f25c2b4e6412'} 2024-03-08T23:44:51.049 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'sha1': 'distro'}} 2024-03-08T23:44:51.049 INFO:teuthology.task.kernel:config {'client.0': {'sha1': 'distro'}}, timeout 300 2024-03-08T23:44:51.050 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T23:44:51.066 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:44:51.067 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-03-08T23:44:51.125 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-361.el9.x86_64 2024-03-08T23:44:51.125 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.14.0-361.el9.x86_64 2024-03-08T23:44:51.125 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum install -y kernel 2024-03-08T23:44:53.233 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-08T23:44:54.816 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 4.2 MB/s | 2.0 MB 00:00 2024-03-08T23:44:55.690 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 15 MB/s | 8.0 MB 00:00 2024-03-08T23:44:58.816 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 9.2 MB/s | 19 MB 00:02 2024-03-08T23:45:03.700 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-08T23:45:05.351 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 15 kB 00:00 2024-03-08T23:45:07.263 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 11 MB/s | 21 MB 00:01 2024-03-08T23:45:12.210 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 25 kB/s | 1.7 kB 00:00 2024-03-08T23:45:14.392 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-08T23:45:14.392 INFO:teuthology.orchestra.run.smithi073.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-08T23:45:14.483 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-03-08T23:45:14.488 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: Package Architecture Version Repository Size 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: kernel x86_64 5.14.0-427.el9 baseos 6.0 M 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core x86_64 5.14.0-427.el9 baseos 21 M 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules x86_64 5.14.0-427.el9 baseos 39 M 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core x86_64 5.14.0-427.el9 baseos 33 M 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:================================================================================ 2024-03-08T23:45:14.489 INFO:teuthology.orchestra.run.smithi073.stdout:Install 4 Packages 2024-03-08T23:45:14.490 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:45:14.490 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 100 M 2024-03-08T23:45:14.490 INFO:teuthology.orchestra.run.smithi073.stdout:Installed size: 123 M 2024-03-08T23:45:14.490 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-03-08T23:45:18.003 INFO:teuthology.orchestra.run.smithi073.stdout:(1/4): kernel-5.14.0-427.el9.x86_64.rpm 1.8 MB/s | 6.0 MB 00:03 2024-03-08T23:45:19.619 INFO:teuthology.orchestra.run.smithi073.stdout:(2/4): kernel-core-5.14.0-427.el9.x86_64.rpm 4.3 MB/s | 21 MB 00:04 2024-03-08T23:45:22.403 INFO:teuthology.orchestra.run.smithi073.stdout:(3/4): kernel-modules-5.14.0-427.el9.x86_64.rpm 5.1 MB/s | 39 MB 00:07 2024-03-08T23:45:25.453 INFO:teuthology.orchestra.run.smithi073.stdout:(4/4): kernel-modules-core-5.14.0-427.el9.x86_6 4.5 MB/s | 33 MB 00:07 2024-03-08T23:45:25.453 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-03-08T23:45:25.453 INFO:teuthology.orchestra.run.smithi073.stdout:Total 9.1 MB/s | 100 MB 00:10 2024-03-08T23:45:26.337 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-03-08T23:45:26.727 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-03-08T23:45:26.727 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-03-08T23:45:28.109 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-03-08T23:45:28.110 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-03-08T23:45:31.088 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-03-08T23:45:31.757 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-core-5.14.0-427.el9.x86_64 1/4 2024-03-08T23:45:31.928 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T23:45:34.782 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T23:45:34.971 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T23:45:39.695 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T23:45:39.823 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:45:44.568 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:46:13.521 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:46:13.521 INFO:teuthology.orchestra.run.smithi073.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-08T23:46:13.521 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:46:13.605 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-modules-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:46:13.973 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: kernel-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:46:13.973 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-5.14.0-427.el9.x86_64 1/4 2024-03-08T23:46:13.973 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-core-5.14.0-427.el9.x86_64 2/4 2024-03-08T23:46:13.973 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-5.14.0-427.el9.x86_64 3/4 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : kernel-modules-core-5.14.0-427.el9.x86_64 4/4 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-5.14.0-427.el9.x86_64 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-core-5.14.0-427.el9.x86_64 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-5.14.0-427.el9.x86_64 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: kernel-modules-core-5.14.0-427.el9.x86_64 2024-03-08T23:46:14.413 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:46:14.414 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-03-08T23:46:14.934 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T23:46:15.084 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T23:46:15.084 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T23:46:15.084 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-427.el9.x86_64 2024-03-08T23:46:15.085 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-08T23:46:15.085 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-08T23:46:15.085 INFO:teuthology.task.kernel:Installing distro kernel on client.0... 2024-03-08T23:46:15.085 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2024-03-08T23:46:15.085 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q kernel | sort -rV | head -n 1 2024-03-08T23:46:15.206 INFO:teuthology.orchestra.run.smithi073.stdout:kernel-5.14.0-427.el9.x86_64 2024-03-08T23:46:15.207 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-427.el9.x86_64 2024-03-08T23:46:15.207 DEBUG:teuthology.orchestra.run.smithi073:> sudo rpm -qi grub2-tools 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Name : grub2-tools 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Epoch : 1 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Version : 2.06 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Release : 68.el9 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: x86_64 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Group : Unspecified 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Size : 8264254 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:License : GPLv3+ 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-08T23:46:15.246 INFO:teuthology.orchestra.run.smithi073.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Packager : builder@centos.org 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Vendor : CentOS 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Summary : Support tools for GRUB. 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:Description : 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:hardware devices. 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:46:15.247 INFO:teuthology.orchestra.run.smithi073.stdout:This subpackage provides tools for support of all platforms. 2024-03-08T23:46:15.248 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-08T23:46:15.248 INFO:teuthology.task.kernel:Updating grub on smithi073 to boot 5.14.0-427.el9.x86_64 2024-03-08T23:46:15.249 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-08T23:46:15.824 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2024-03-08T23:46:17.342 INFO:teuthology.orchestra.run.smithi073.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-08T23:46:17.375 INFO:teuthology.orchestra.run.smithi073.stderr:done 2024-03-08T23:46:17.378 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-03-08T23:46:17.434 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.QtW9gI14G9 2024-03-08T23:46:17.434 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.QtW9gI14G9 2024-03-08T23:46:17.504 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.QtW9gI14G9 2024-03-08T23:46:17.647 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.QtW9gI14G9 is 6KB 2024-03-08T23:46:17.697 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.QtW9gI14G9 2024-03-08T23:46:17.711 DEBUG:teuthology.orchestra.run.smithi073:> sudo /bin/ls /boot/loader/entries || true 2024-03-08T23:46:17.775 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-08T23:46:17.776 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-08T23:46:17.776 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-08T23:46:17.776 INFO:teuthology.orchestra.run.smithi073.stdout:15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64.conf 2024-03-08T23:46:17.777 DEBUG:teuthology.orchestra.run.smithi073:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-427.el9.x86_64 2024-03-08T23:46:17.920 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2024-03-08T23:46:47.922 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-08T23:46:47.923 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:46:47.924 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:47:06.398 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-03-08T23:47:15.399 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:47:15.400 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:47:18.462 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-03-08T23:47:30.463 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:47:30.464 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:48:02.275 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-03-08T23:48:02.730 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:48:02.731 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5.14.0-427.el9.x86_64"... 2024-03-08T23:48:02.731 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-03-08T23:48:02.745 INFO:teuthology.orchestra.run.smithi073.stdout:5.14.0-427.el9.x86_64 2024-03-08T23:48:02.745 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-427.el9.x86_64 vs 5.14.0-427.el9.x86_64 2024-03-08T23:48:02.745 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-08T23:48:02.746 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-08T23:48:03.747 DEBUG:teuthology.parallel:result is None 2024-03-08T23:48:03.747 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-08T23:48:03.755 INFO:teuthology.task.internal:Creating test directory... 2024-03-08T23:48:03.755 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T23:48:03.758 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T23:48:03.761 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-08T23:48:03.778 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-08T23:48:03.785 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-08T23:48:03.792 INFO:teuthology.task.internal:Creating archive directory... 2024-03-08T23:48:03.792 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T23:48:03.809 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T23:48:03.818 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-08T23:48:04.151 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-08T23:48:04.158 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-08T23:48:04.159 DEBUG:teuthology.orchestra.run.smithi102:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T23:48:04.175 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:48:04.175 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-08T23:48:04.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:48:04.192 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T23:48:04.194 DEBUG:teuthology.orchestra.run.smithi102:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T23:48:04.218 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-08T23:48:04.260 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.271 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.274 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.295 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.296 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.315 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-08T23:48:04.317 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-08T23:48:04.324 INFO:teuthology.task.internal:Configuring sudo... 2024-03-08T23:48:04.325 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T23:48:04.327 DEBUG:teuthology.orchestra.run.smithi102:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T23:48:04.330 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-08T23:48:04.367 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-08T23:48:04.377 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-08T23:48:04.377 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T23:48:04.390 DEBUG:teuthology.orchestra.run.smithi102:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T23:48:04.399 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-08T23:48:04.423 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:04.483 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:04.559 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:04.602 DEBUG:teuthology.orchestra.run.smithi073:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:04.638 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:48:04.638 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T23:48:04.702 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:04.740 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:04.800 DEBUG:teuthology.orchestra.run.smithi102:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:04.883 DEBUG:teuthology.orchestra.run.smithi102:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:04.926 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:48:04.927 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T23:48:04.995 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:05.039 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-08T23:48:05.095 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:05.182 DEBUG:teuthology.orchestra.run.smithi179:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-08T23:48:05.226 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:48:05.226 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-08T23:48:05.295 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-03-08T23:48:05.297 DEBUG:teuthology.orchestra.run.smithi102:> sudo service rsyslog restart 2024-03-08T23:48:05.300 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2024-03-08T23:48:05.341 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:48:05.356 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:48:05.376 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-08T23:48:05.860 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-08T23:48:05.868 INFO:teuthology.task.internal:Starting timer... 2024-03-08T23:48:05.868 INFO:teuthology.run_tasks:Running task pcp... 2024-03-08T23:48:05.879 INFO:teuthology.run_tasks:Running task selinux... 2024-03-08T23:48:05.887 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-08T23:48:05.888 DEBUG:teuthology.orchestra.run.smithi073:> sudo service auditd rotate 2024-03-08T23:48:05.963 INFO:teuthology.orchestra.run.smithi073.stdout:Rotating logs: 2024-03-08T23:48:05.966 DEBUG:teuthology.orchestra.run.smithi102:> sudo service auditd rotate 2024-03-08T23:48:06.024 INFO:teuthology.orchestra.run.smithi102.stdout:Rotating logs: 2024-03-08T23:48:06.025 DEBUG:teuthology.orchestra.run.smithi179:> sudo service auditd rotate 2024-03-08T23:48:06.084 INFO:teuthology.orchestra.run.smithi179.stdout:Rotating logs: 2024-03-08T23:48:06.086 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-08T23:48:06.087 DEBUG:teuthology.orchestra.run.smithi073:> /usr/sbin/getenforce 2024-03-08T23:48:06.112 INFO:teuthology.orchestra.run.smithi073.stdout:Permissive 2024-03-08T23:48:06.112 DEBUG:teuthology.orchestra.run.smithi102:> /usr/sbin/getenforce 2024-03-08T23:48:06.136 INFO:teuthology.orchestra.run.smithi102.stdout:Permissive 2024-03-08T23:48:06.136 DEBUG:teuthology.orchestra.run.smithi179:> /usr/sbin/getenforce 2024-03-08T23:48:06.163 INFO:teuthology.orchestra.run.smithi179.stdout:Permissive 2024-03-08T23:48:06.163 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi073.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi102.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi179.front.sepia.ceph.com': 'permissive'} 2024-03-08T23:48:06.164 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:48:06.191 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:48:06.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:48:06.220 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:48:06.220 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-08T23:48:06.250 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-08T23:48:06.250 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-08T23:48:06.250 DEBUG:teuthology.orchestra.run.smithi073:> sudo /usr/sbin/setenforce permissive 2024-03-08T23:48:06.280 DEBUG:teuthology.orchestra.run.smithi102:> sudo /usr/sbin/setenforce permissive 2024-03-08T23:48:06.312 DEBUG:teuthology.orchestra.run.smithi179:> sudo /usr/sbin/setenforce permissive 2024-03-08T23:48:06.348 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-08T23:48:06.358 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-08T23:48:06.367 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-08T23:48:06.499 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-08T23:48:06.513 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-08T23:48:06.514 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com,smithi102.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-08T23:52:39.743 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com'), Remote(name='ubuntu@smithi102.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2024-03-08T23:52:39.744 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:52:39.745 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:52:39.833 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-03-08T23:52:39.909 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-03-08T23:52:39.909 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-08T23:52:39.910 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi102.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:52:39.995 DEBUG:teuthology.orchestra.run.smithi102:> true 2024-03-08T23:52:40.066 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi102.front.sepia.ceph.com' 2024-03-08T23:52:40.066 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-08T23:52:40.067 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-08T23:52:40.155 DEBUG:teuthology.orchestra.run.smithi179:> true 2024-03-08T23:52:40.221 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi179.front.sepia.ceph.com' 2024-03-08T23:52:40.222 INFO:teuthology.run_tasks:Running task clock... 2024-03-08T23:52:40.233 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-08T23:52:40.233 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T23:52:40.234 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:52:40.236 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T23:52:40.236 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:52:40.239 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-08T23:52:40.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-08T23:52:40.265 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T23:52:40.269 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T23:52:40.282 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T23:52:40.287 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T23:52:40.293 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-08T23:52:40.311 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-08T23:52:40.311 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: ntpd: command not found 2024-03-08T23:52:40.317 INFO:teuthology.orchestra.run.smithi102.stderr:sudo: ntpd: command not found 2024-03-08T23:52:40.324 INFO:teuthology.orchestra.run.smithi073.stdout:506 Cannot talk to daemon 2024-03-08T23:52:40.331 INFO:teuthology.orchestra.run.smithi102.stdout:506 Cannot talk to daemon 2024-03-08T23:52:40.340 INFO:teuthology.orchestra.run.smithi179.stderr:sudo: ntpd: command not found 2024-03-08T23:52:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T23:52:40.348 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T23:52:40.354 INFO:teuthology.orchestra.run.smithi179.stdout:506 Cannot talk to daemon 2024-03-08T23:52:40.357 INFO:teuthology.orchestra.run.smithi073.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T23:52:40.364 INFO:teuthology.orchestra.run.smithi102.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T23:52:40.371 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-08T23:52:40.387 INFO:teuthology.orchestra.run.smithi179.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-08T23:52:40.406 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:52:40.408 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.410 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.411 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:52:40.411 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-08T23:52:40.411 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.411 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.412 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.412 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.430 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.433 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-08T23:52:40.434 INFO:teuthology.run_tasks:Running task install... 2024-03-08T23:52:40.442 DEBUG:teuthology.task.install:project ceph 2024-03-08T23:52:40.442 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '7d63482aa89628d733e298984a762389b625c761'}} 2024-03-08T23:52:40.443 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '7d63482aa89628d733e298984a762389b625c761'} 2024-03-08T23:52:40.443 INFO:teuthology.task.install:Using flavor: default 2024-03-08T23:52:40.450 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-08T23:52:40.450 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-08T23:52:40.450 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T23:52:40.451 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:52:40.452 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T23:52:40.453 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:52:40.454 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'tag': None, 'wait_for_package': False} 2024-03-08T23:52:40.454 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:52:40.676 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T23:52:40.676 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T23:52:40.687 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T23:52:40.687 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T23:52:40.693 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/ 2024-03-08T23:52:40.693 INFO:teuthology.task.install.rpm:Package version is 19.0.0-1965.g7d63482a 2024-03-08T23:52:40.807 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T23:52:40.807 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:52:40.807 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T23:52:40.835 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T23:52:40.835 DEBUG:teuthology.orchestra.run.smithi179:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T23:52:40.844 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T23:52:40.844 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:52:40.844 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T23:52:40.857 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-08T23:52:40.858 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:52:40.858 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-08T23:52:40.873 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T23:52:40.873 DEBUG:teuthology.orchestra.run.smithi073:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T23:52:40.886 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-08T23:52:40.886 DEBUG:teuthology.orchestra.run.smithi102:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/7d63482aa89628d733e298984a762389b625c761/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-08T23:52:40.903 DEBUG:teuthology.orchestra.run.smithi179:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T23:52:40.941 DEBUG:teuthology.orchestra.run.smithi073:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T23:52:40.955 DEBUG:teuthology.orchestra.run.smithi102:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-08T23:52:40.983 DEBUG:teuthology.orchestra.run.smithi179:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T23:52:41.023 DEBUG:teuthology.orchestra.run.smithi073:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T23:52:41.035 DEBUG:teuthology.orchestra.run.smithi102:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-08T23:52:41.049 INFO:teuthology.orchestra.run.smithi179.stdout:check_obsoletes = 1 2024-03-08T23:52:41.051 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum clean all 2024-03-08T23:52:41.089 INFO:teuthology.orchestra.run.smithi073.stdout:check_obsoletes = 1 2024-03-08T23:52:41.091 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum clean all 2024-03-08T23:52:41.102 INFO:teuthology.orchestra.run.smithi102.stdout:check_obsoletes = 1 2024-03-08T23:52:41.103 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum clean all 2024-03-08T23:52:41.284 INFO:teuthology.orchestra.run.smithi179.stdout:74 files removed 2024-03-08T23:52:41.311 DEBUG:teuthology.orchestra.run.smithi179:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T23:52:41.328 INFO:teuthology.orchestra.run.smithi073.stdout:74 files removed 2024-03-08T23:52:41.336 INFO:teuthology.orchestra.run.smithi102.stdout:74 files removed 2024-03-08T23:52:41.354 DEBUG:teuthology.orchestra.run.smithi073:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T23:52:41.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-08T23:52:41.921 INFO:teuthology.orchestra.run.smithi179.stdout:ceph packages for x86_64 255 kB/s | 82 kB 00:00 2024-03-08T23:52:42.058 INFO:teuthology.orchestra.run.smithi102.stdout:ceph packages for x86_64 202 kB/s | 82 kB 00:00 2024-03-08T23:52:42.133 INFO:teuthology.orchestra.run.smithi073.stdout:ceph packages for x86_64 177 kB/s | 82 kB 00:00 2024-03-08T23:52:42.271 INFO:teuthology.orchestra.run.smithi179.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-08T23:52:42.409 INFO:teuthology.orchestra.run.smithi102.stdout:ceph noarch packages 33 kB/s | 10 kB 00:00 2024-03-08T23:52:42.501 INFO:teuthology.orchestra.run.smithi073.stdout:ceph noarch packages 32 kB/s | 10 kB 00:00 2024-03-08T23:52:42.647 INFO:teuthology.orchestra.run.smithi179.stdout:ceph source packages 5.5 kB/s | 1.9 kB 00:00 2024-03-08T23:52:42.801 INFO:teuthology.orchestra.run.smithi073.stdout:ceph source packages 7.2 kB/s | 1.9 kB 00:00 2024-03-08T23:52:42.809 INFO:teuthology.orchestra.run.smithi102.stdout:ceph source packages 5.1 kB/s | 1.9 kB 00:00 2024-03-08T23:52:43.049 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-08T23:52:43.245 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-08T23:52:43.278 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-08T23:52:44.591 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS-9 - BaseOS 4.5 MB/s | 2.0 MB 00:00 2024-03-08T23:52:44.687 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS-9 - BaseOS 6.6 MB/s | 2.0 MB 00:00 2024-03-08T23:52:44.712 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS-9 - BaseOS 6.0 MB/s | 2.0 MB 00:00 2024-03-08T23:52:45.888 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - BaseOS 9.5 MB/s | 8.0 MB 00:00 2024-03-08T23:52:45.965 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - BaseOS 7.7 MB/s | 8.0 MB 00:01 2024-03-08T23:52:46.679 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - BaseOS 4.8 MB/s | 8.0 MB 00:01 2024-03-08T23:52:50.108 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - AppStream 6.1 MB/s | 19 MB 00:03 2024-03-08T23:52:54.384 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-08T23:52:55.826 INFO:teuthology.orchestra.run.smithi179.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 15 kB 00:00 2024-03-08T23:52:56.936 INFO:teuthology.orchestra.run.smithi179.stdout:Extra Packages for Enterprise Linux 21 MB/s | 21 MB 00:00 2024-03-08T23:53:01.786 INFO:teuthology.orchestra.run.smithi179.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-08T23:53:04.232 INFO:teuthology.orchestra.run.smithi179.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:04.233 INFO:teuthology.orchestra.run.smithi179.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:04.247 INFO:teuthology.orchestra.run.smithi179.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T23:53:04.248 INFO:teuthology.orchestra.run.smithi179.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T23:53:04.249 INFO:teuthology.orchestra.run.smithi179.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T23:53:04.250 INFO:teuthology.orchestra.run.smithi179.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:04.252 INFO:teuthology.orchestra.run.smithi179.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:04.255 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T23:53:04.256 INFO:teuthology.orchestra.run.smithi179.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T23:53:04.334 INFO:teuthology.orchestra.run.smithi179.stdout:Dependencies resolved. 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: Package Arch Version Repository Size 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout:Installing: 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T23:53:04.346 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T23:53:04.347 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T23:53:04.348 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrading: 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T23:53:04.349 INFO:teuthology.orchestra.run.smithi179.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T23:53:04.350 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout:Installing dependencies: 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T23:53:04.351 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T23:53:04.352 INFO:teuthology.orchestra.run.smithi179.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T23:53:04.353 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T23:53:04.354 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T23:53:04.355 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T23:53:04.356 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T23:53:04.357 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T23:53:04.358 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Installing weak dependencies: 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:53:04.359 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction Summary 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================================ 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Install 150 Packages 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Upgrade 24 Packages 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Total download size: 223 M 2024-03-08T23:53:04.360 INFO:teuthology.orchestra.run.smithi179.stdout:Downloading Packages: 2024-03-08T23:53:05.303 INFO:teuthology.orchestra.run.smithi179.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 41 kB/s | 6.6 kB 00:00 2024-03-08T23:53:05.487 INFO:teuthology.orchestra.run.smithi179.stdout:(2/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 4.7 MB/s | 889 kB 00:00 2024-03-08T23:53:05.554 INFO:teuthology.orchestra.run.smithi179.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 2.1 MB/s | 145 kB 00:00 2024-03-08T23:53:05.838 INFO:teuthology.orchestra.run.smithi179.stdout:(4/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 7.5 MB/s | 2.1 MB 00:00 2024-03-08T23:53:06.063 INFO:teuthology.orchestra.run.smithi179.stdout:(5/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 6.6 MB/s | 1.5 MB 00:00 2024-03-08T23:53:06.605 INFO:teuthology.orchestra.run.smithi179.stdout:(6/174): ceph-common-19.0.0-1965.g7d63482a.el9. 14 MB/s | 20 MB 00:01 2024-03-08T23:53:06.680 INFO:teuthology.orchestra.run.smithi179.stdout:(7/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 7.6 MB/s | 4.7 MB 00:00 2024-03-08T23:53:07.764 INFO:teuthology.orchestra.run.smithi179.stdout:(8/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x86 15 MB/s | 17 MB 00:01 2024-03-08T23:53:07.881 INFO:teuthology.orchestra.run.smithi179.stdout:(9/174): ceph-radosgw-19.0.0-1965.g7d63482a.el9 8.2 MB/s | 9.8 MB 00:01 2024-03-08T23:53:07.906 INFO:teuthology.orchestra.run.smithi179.stdout:(10/174): ceph-selinux-19.0.0-1965.g7d63482a.el 177 kB/s | 25 kB 00:00 2024-03-08T23:53:07.972 INFO:teuthology.orchestra.run.smithi179.stdout:(11/174): ceph-base-19.0.0-1965.g7d63482a.el9.x 1.9 MB/s | 5.4 MB 00:02 2024-03-08T23:53:07.998 INFO:teuthology.orchestra.run.smithi179.stdout:(12/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 2.3 MB/s | 220 kB 00:00 2024-03-08T23:53:08.040 INFO:teuthology.orchestra.run.smithi179.stdout:(13/174): cephfs-shell-19.0.0-1965.g7d63482a.el 404 kB/s | 27 kB 00:00 2024-03-08T23:53:08.065 INFO:teuthology.orchestra.run.smithi179.stdout:(14/174): libcephfs-devel-19.0.0-1965.g7d63482a 464 kB/s | 31 kB 00:00 2024-03-08T23:53:08.123 INFO:teuthology.orchestra.run.smithi179.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 2.7 MB/s | 163 kB 00:00 2024-03-08T23:53:08.182 INFO:teuthology.orchestra.run.smithi179.stdout:(16/174): librados-devel-19.0.0-1965.g7d63482a. 2.1 MB/s | 127 kB 00:00 2024-03-08T23:53:08.266 INFO:teuthology.orchestra.run.smithi179.stdout:(17/174): libradosstriper1-19.0.0-1965.g7d63482 5.9 MB/s | 501 kB 00:00 2024-03-08T23:53:08.300 INFO:teuthology.orchestra.run.smithi179.stdout:(18/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 2.8 MB/s | 745 kB 00:00 2024-03-08T23:53:08.400 INFO:teuthology.orchestra.run.smithi179.stdout:(19/174): python3-ceph-argparse-19.0.0-1965.g7d 452 kB/s | 45 kB 00:00 2024-03-08T23:53:08.509 INFO:teuthology.orchestra.run.smithi179.stdout:(20/174): python3-ceph-common-19.0.0-1965.g7d63 1.2 MB/s | 131 kB 00:00 2024-03-08T23:53:08.651 INFO:teuthology.orchestra.run.smithi179.stdout:(21/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 13 MB/s | 5.1 MB 00:00 2024-03-08T23:53:08.676 INFO:teuthology.orchestra.run.smithi179.stdout:(22/174): python3-cephfs-19.0.0-1965.g7d63482a. 971 kB/s | 162 kB 00:00 2024-03-08T23:53:08.718 INFO:teuthology.orchestra.run.smithi179.stdout:(23/174): python3-rados-19.0.0-1965.g7d63482a.e 4.7 MB/s | 320 kB 00:00 2024-03-08T23:53:08.776 INFO:teuthology.orchestra.run.smithi179.stdout:(24/174): python3-rgw-19.0.0-1965.g7d63482a.el9 1.7 MB/s | 99 kB 00:00 2024-03-08T23:53:08.802 INFO:teuthology.orchestra.run.smithi179.stdout:(25/174): python3-rbd-19.0.0-1965.g7d63482a.el9 2.3 MB/s | 298 kB 00:00 2024-03-08T23:53:08.835 INFO:teuthology.orchestra.run.smithi179.stdout:(26/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-08T23:53:08.894 INFO:teuthology.orchestra.run.smithi179.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-08T23:53:08.952 INFO:teuthology.orchestra.run.smithi179.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1965.g 581 kB/s | 34 kB 00:00 2024-03-08T23:53:09.028 INFO:teuthology.orchestra.run.smithi179.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 1.8 MB/s | 139 kB 00:00 2024-03-08T23:53:09.070 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - AppStream 872 kB/s | 19 MB 00:22 2024-03-08T23:53:09.203 INFO:teuthology.orchestra.run.smithi179.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 10 MB/s | 1.8 MB 00:00 2024-03-08T23:53:09.778 INFO:teuthology.orchestra.run.smithi179.stdout:(31/174): ceph-mgr-diskprediction-local-19.0.0- 13 MB/s | 7.4 MB 00:00 2024-03-08T23:53:09.862 INFO:teuthology.orchestra.run.smithi179.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1965.g7d 2.9 MB/s | 248 kB 00:00 2024-03-08T23:53:09.920 INFO:teuthology.orchestra.run.smithi179.stdout:(33/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 846 kB/s | 49 kB 00:00 2024-03-08T23:53:09.971 INFO:teuthology.orchestra.run.smithi179.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1965.g7 330 kB/s | 16 kB 00:00 2024-03-08T23:53:10.030 INFO:teuthology.orchestra.run.smithi179.stdout:(35/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 2.5 MB/s | 3.1 MB 00:01 2024-03-08T23:53:10.055 INFO:teuthology.orchestra.run.smithi179.stdout:(36/174): ceph-volume-19.0.0-1965.g7d63482a.el9 3.1 MB/s | 264 kB 00:00 2024-03-08T23:53:10.105 INFO:teuthology.orchestra.run.smithi179.stdout:(37/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 367 kB/s | 18 kB 00:00 2024-03-08T23:53:10.248 INFO:teuthology.orchestra.run.smithi179.stdout:(38/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 3.4 MB/s | 748 kB 00:00 2024-03-08T23:53:10.272 INFO:teuthology.orchestra.run.smithi179.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-08T23:53:10.314 INFO:teuthology.orchestra.run.smithi179.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 14 MB/s | 920 kB 00:00 2024-03-08T23:53:10.340 INFO:teuthology.orchestra.run.smithi179.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-08T23:53:10.365 INFO:teuthology.orchestra.run.smithi179.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-08T23:53:10.382 INFO:teuthology.orchestra.run.smithi179.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 768 kB/s | 32 kB 00:00 2024-03-08T23:53:10.465 INFO:teuthology.orchestra.run.smithi179.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 30 MB/s | 3.0 MB 00:00 2024-03-08T23:53:10.482 INFO:teuthology.orchestra.run.smithi179.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 172 kB/s | 17 kB 00:00 2024-03-08T23:53:10.507 INFO:teuthology.orchestra.run.smithi179.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 242 kB/s | 10 kB 00:00 2024-03-08T23:53:10.533 INFO:teuthology.orchestra.run.smithi179.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-08T23:53:10.558 INFO:teuthology.orchestra.run.smithi179.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-08T23:53:10.583 INFO:teuthology.orchestra.run.smithi179.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-08T23:53:10.609 INFO:teuthology.orchestra.run.smithi179.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-08T23:53:10.634 INFO:teuthology.orchestra.run.smithi179.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-08T23:53:10.668 INFO:teuthology.orchestra.run.smithi179.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-08T23:53:10.693 INFO:teuthology.orchestra.run.smithi179.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-08T23:53:10.718 INFO:teuthology.orchestra.run.smithi179.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.3 MB/s | 217 kB 00:00 2024-03-08T23:53:10.744 INFO:teuthology.orchestra.run.smithi179.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-08T23:53:11.177 INFO:teuthology.orchestra.run.smithi179.stdout:(56/174): ceph-test-19.0.0-1965.g7d63482a.el9.x 14 MB/s | 47 MB 00:03 2024-03-08T23:53:11.249 INFO:teuthology.orchestra.run.smithi179.stdout:(57/174): openblas-openmp-0.3.21-2.el9.x86_64.r 8.9 MB/s | 4.7 MB 00:00 2024-03-08T23:53:11.291 INFO:teuthology.orchestra.run.smithi179.stdout:(58/174): postgresql-13.11-1.el9.x86_64.rpm 2.8 MB/s | 1.5 MB 00:00 2024-03-08T23:53:11.324 INFO:teuthology.orchestra.run.smithi179.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 5.5 MB/s | 811 kB 00:00 2024-03-08T23:53:11.358 INFO:teuthology.orchestra.run.smithi179.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 1.2 MB/s | 134 kB 00:00 2024-03-08T23:53:11.399 INFO:teuthology.orchestra.run.smithi179.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 14 MB/s | 1.0 MB 00:00 2024-03-08T23:53:11.501 INFO:teuthology.orchestra.run.smithi179.stdout:(62/174): postgresql-server-13.11-1.el9.x86_64. 32 MB/s | 5.7 MB 00:00 2024-03-08T23:53:11.526 INFO:teuthology.orchestra.run.smithi179.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 463 kB/s | 78 kB 00:00 2024-03-08T23:53:11.609 INFO:teuthology.orchestra.run.smithi179.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 28 MB/s | 5.8 MB 00:00 2024-03-08T23:53:11.643 INFO:teuthology.orchestra.run.smithi179.stdout:(65/174): python3-jsonpatch-1.21-16.el9.noarch. 175 kB/s | 24 kB 00:00 2024-03-08T23:53:11.668 INFO:teuthology.orchestra.run.smithi179.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 129 kB/s | 18 kB 00:00 2024-03-08T23:53:11.693 INFO:teuthology.orchestra.run.smithi179.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.7 MB/s | 144 kB 00:00 2024-03-08T23:53:11.718 INFO:teuthology.orchestra.run.smithi179.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 635 kB/s | 32 kB 00:00 2024-03-08T23:53:11.803 INFO:teuthology.orchestra.run.smithi179.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 36 MB/s | 4.8 MB 00:00 2024-03-08T23:53:11.828 INFO:teuthology.orchestra.run.smithi179.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.6 MB/s | 217 kB 00:00 2024-03-08T23:53:11.853 INFO:teuthology.orchestra.run.smithi179.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.3 MB/s | 174 kB 00:00 2024-03-08T23:53:11.878 INFO:teuthology.orchestra.run.smithi179.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 922 kB/s | 69 kB 00:00 2024-03-08T23:53:11.903 INFO:teuthology.orchestra.run.smithi179.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 822 kB/s | 41 kB 00:00 2024-03-08T23:53:11.928 INFO:teuthology.orchestra.run.smithi179.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.1 MB/s | 238 kB 00:00 2024-03-08T23:53:11.954 INFO:teuthology.orchestra.run.smithi179.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-08T23:53:11.970 INFO:teuthology.orchestra.run.smithi179.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-08T23:53:11.996 INFO:teuthology.orchestra.run.smithi179.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.1 MB/s | 211 kB 00:00 2024-03-08T23:53:12.013 INFO:teuthology.orchestra.run.smithi179.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 728 kB/s | 43 kB 00:00 2024-03-08T23:53:12.346 INFO:teuthology.orchestra.run.smithi179.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 42 MB/s | 16 MB 00:00 2024-03-08T23:53:12.363 INFO:teuthology.orchestra.run.smithi179.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 118 kB/s | 41 kB 00:00 2024-03-08T23:53:12.380 INFO:teuthology.orchestra.run.smithi179.stdout:(81/174): python3-toml-0.10.2-6.el9.noarch.rpm 106 kB/s | 39 kB 00:00 2024-03-08T23:53:12.405 INFO:teuthology.orchestra.run.smithi179.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 5.0 MB/s | 300 kB 00:00 2024-03-08T23:53:12.422 INFO:teuthology.orchestra.run.smithi179.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 967 kB/s | 56 kB 00:00 2024-03-08T23:53:12.439 INFO:teuthology.orchestra.run.smithi179.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-08T23:53:12.456 INFO:teuthology.orchestra.run.smithi179.stdout:(85/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-08T23:53:12.473 INFO:teuthology.orchestra.run.smithi179.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-08T23:53:12.498 INFO:teuthology.orchestra.run.smithi179.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.0 MB/s | 241 kB 00:00 2024-03-08T23:53:12.523 INFO:teuthology.orchestra.run.smithi179.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-08T23:53:12.565 INFO:teuthology.orchestra.run.smithi179.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-08T23:53:12.590 INFO:teuthology.orchestra.run.smithi179.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-08T23:53:12.616 INFO:teuthology.orchestra.run.smithi179.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.0 MB/s | 187 kB 00:00 2024-03-08T23:53:12.750 INFO:teuthology.orchestra.run.smithi179.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.2 MB/s | 187 kB 00:00 2024-03-08T23:53:12.790 INFO:teuthology.orchestra.run.smithi179.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.1 MB/s | 195 kB 00:00 2024-03-08T23:53:12.808 INFO:teuthology.orchestra.run.smithi179.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-08T23:53:12.842 INFO:teuthology.orchestra.run.smithi179.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-03-08T23:53:12.884 INFO:teuthology.orchestra.run.smithi179.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 229 kB/s | 21 kB 00:00 2024-03-08T23:53:12.901 INFO:teuthology.orchestra.run.smithi179.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-08T23:53:13.001 INFO:teuthology.orchestra.run.smithi179.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.1 MB/s | 245 kB 00:00 2024-03-08T23:53:13.026 INFO:teuthology.orchestra.run.smithi179.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.9 MB/s | 248 kB 00:00 2024-03-08T23:53:13.068 INFO:teuthology.orchestra.run.smithi179.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 836 kB/s | 35 kB 00:00 2024-03-08T23:53:13.094 INFO:teuthology.orchestra.run.smithi179.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.3 MB/s | 216 kB 00:00 2024-03-08T23:53:13.119 INFO:teuthology.orchestra.run.smithi179.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-08T23:53:13.161 INFO:teuthology.orchestra.run.smithi179.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 537 kB/s | 22 kB 00:00 2024-03-08T23:53:13.261 INFO:teuthology.orchestra.run.smithi179.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 561 kB/s | 94 kB 00:00 2024-03-08T23:53:13.303 INFO:teuthology.orchestra.run.smithi179.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 5.9 MB/s | 863 kB 00:00 2024-03-08T23:53:13.969 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - CRB 4.5 MB/s | 6.1 MB 00:01 2024-03-08T23:53:14.320 INFO:teuthology.orchestra.run.smithi179.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 2.8 MB/s | 4.1 MB 00:01 2024-03-08T23:53:14.354 INFO:teuthology.orchestra.run.smithi179.stdout:(107/174): gperftools-libs-2.9.1-2.el9.x86_64.r 294 kB/s | 309 kB 00:01 2024-03-08T23:53:14.437 INFO:teuthology.orchestra.run.smithi179.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 167 kB/s | 19 kB 00:00 2024-03-08T23:53:14.554 INFO:teuthology.orchestra.run.smithi179.stdout:(109/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 223 kB/s | 26 kB 00:00 2024-03-08T23:53:14.679 INFO:teuthology.orchestra.run.smithi179.stdout:(110/174): abseil-cpp-20211102.0-3.el9.x86_64.r 387 kB/s | 548 kB 00:01 2024-03-08T23:53:14.763 INFO:teuthology.orchestra.run.smithi179.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 233 kB/s | 49 kB 00:00 2024-03-08T23:53:14.888 INFO:teuthology.orchestra.run.smithi179.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 323 kB/s | 67 kB 00:00 2024-03-08T23:53:15.147 INFO:teuthology.orchestra.run.smithi179.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 394 kB/s | 151 kB 00:00 2024-03-08T23:53:15.945 INFO:teuthology.orchestra.run.smithi073.stdout:CentOS Stream 9 - Extras packages 26 kB/s | 15 kB 00:00 2024-03-08T23:53:16.172 INFO:teuthology.orchestra.run.smithi179.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 534 kB/s | 548 kB 00:01 2024-03-08T23:53:16.292 INFO:teuthology.orchestra.run.smithi179.stdout:(115/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 363 kB/s | 43 kB 00:00 2024-03-08T23:53:16.317 INFO:teuthology.orchestra.run.smithi179.stdout:(116/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 587 kB/s | 838 kB 00:01 2024-03-08T23:53:16.401 INFO:teuthology.orchestra.run.smithi179.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-08T23:53:16.426 INFO:teuthology.orchestra.run.smithi179.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 138 kB/s | 15 kB 00:00 2024-03-08T23:53:16.868 INFO:teuthology.orchestra.run.smithi179.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 368 kB/s | 172 kB 00:00 2024-03-08T23:53:16.926 INFO:teuthology.orchestra.run.smithi073.stdout:Extra Packages for Enterprise Linux 24 MB/s | 21 MB 00:00 2024-03-08T23:53:17.127 INFO:teuthology.orchestra.run.smithi179.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 512 kB/s | 358 kB 00:00 2024-03-08T23:53:17.235 INFO:teuthology.orchestra.run.smithi179.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 326 kB/s | 35 kB 00:00 2024-03-08T23:53:17.444 INFO:teuthology.orchestra.run.smithi179.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 527 kB/s | 303 kB 00:00 2024-03-08T23:53:17.653 INFO:teuthology.orchestra.run.smithi179.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 486 kB/s | 203 kB 00:00 2024-03-08T23:53:17.953 INFO:teuthology.orchestra.run.smithi179.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 482 kB/s | 144 kB 00:00 2024-03-08T23:53:18.070 INFO:teuthology.orchestra.run.smithi179.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 91 kB/s | 11 kB 00:00 2024-03-08T23:53:18.204 INFO:teuthology.orchestra.run.smithi179.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 132 kB/s | 18 kB 00:00 2024-03-08T23:53:18.362 INFO:teuthology.orchestra.run.smithi179.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 146 kB/s | 23 kB 00:00 2024-03-08T23:53:18.488 INFO:teuthology.orchestra.run.smithi179.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 156 kB/s | 19 kB 00:00 2024-03-08T23:53:18.604 INFO:teuthology.orchestra.run.smithi179.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 169 kB/s | 20 kB 00:00 2024-03-08T23:53:19.410 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - AppStream 610 kB/s | 19 MB 00:31 2024-03-08T23:53:20.230 INFO:teuthology.orchestra.run.smithi179.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 643 kB/s | 1.0 MB 00:01 2024-03-08T23:53:20.355 INFO:teuthology.orchestra.run.smithi179.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 371 kB/s | 46 kB 00:00 2024-03-08T23:53:20.647 INFO:teuthology.orchestra.run.smithi179.stdout:(132/174): python3-grpcio-1.46.7-10.el9.x86_64. 651 kB/s | 2.0 MB 00:03 2024-03-08T23:53:20.672 INFO:teuthology.orchestra.run.smithi179.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 249 kB/s | 79 kB 00:00 2024-03-08T23:53:20.764 INFO:teuthology.orchestra.run.smithi179.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 494 kB/s | 58 kB 00:00 2024-03-08T23:53:20.881 INFO:teuthology.orchestra.run.smithi179.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 141 kB/s | 16 kB 00:00 2024-03-08T23:53:21.065 INFO:teuthology.orchestra.run.smithi179.stdout:(136/174): python3-pecan-1.4.2-3.el9.noarch.rpm 694 kB/s | 272 kB 00:00 2024-03-08T23:53:21.090 INFO:teuthology.orchestra.run.smithi179.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 431 kB/s | 90 kB 00:00 2024-03-08T23:53:21.182 INFO:teuthology.orchestra.run.smithi179.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-08T23:53:21.207 INFO:teuthology.orchestra.run.smithi179.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 263 kB/s | 31 kB 00:00 2024-03-08T23:53:21.324 INFO:teuthology.orchestra.run.smithi179.stdout:(140/174): python3-rsa-4.9-2.el9.noarch.rpm 508 kB/s | 59 kB 00:00 2024-03-08T23:53:21.433 INFO:teuthology.orchestra.run.smithi179.stdout:(141/174): libarrow-9.0.0-10.el9.x86_64.rpm 639 kB/s | 4.4 MB 00:07 2024-03-08T23:53:21.450 INFO:teuthology.orchestra.run.smithi179.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 285 kB/s | 36 kB 00:00 2024-03-08T23:53:21.475 INFO:teuthology.orchestra.run.smithi179.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 644 kB/s | 188 kB 00:00 2024-03-08T23:53:21.542 INFO:teuthology.orchestra.run.smithi179.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 469 kB/s | 51 kB 00:00 2024-03-08T23:53:21.726 INFO:teuthology.orchestra.run.smithi179.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 358 kB/s | 90 kB 00:00 2024-03-08T23:53:21.851 INFO:teuthology.orchestra.run.smithi179.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 573 kB/s | 230 kB 00:00 2024-03-08T23:53:21.882 INFO:teuthology.orchestra.run.smithi073.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-08T23:53:21.926 INFO:teuthology.orchestra.run.smithi179.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 100 kB/s | 20 kB 00:00 2024-03-08T23:53:22.143 INFO:teuthology.orchestra.run.smithi179.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 711 kB/s | 427 kB 00:00 2024-03-08T23:53:22.260 INFO:teuthology.orchestra.run.smithi179.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 574 kB/s | 192 kB 00:00 2024-03-08T23:53:22.435 INFO:teuthology.orchestra.run.smithi179.stdout:(150/174): librados2-19.0.0-1965.g7d63482a.el9. 12 MB/s | 3.4 MB 00:00 2024-03-08T23:53:22.477 INFO:teuthology.orchestra.run.smithi179.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-08T23:53:23.002 INFO:teuthology.orchestra.run.smithi179.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 4.4 MB/s | 2.3 MB 00:00 2024-03-08T23:53:23.061 INFO:teuthology.orchestra.run.smithi179.stdout:(153/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 3.9 MB/s | 3.1 MB 00:00 2024-03-08T23:53:23.144 INFO:teuthology.orchestra.run.smithi179.stdout:(154/174): NetworkManager-team-1.46.0-1.el9.x86 509 kB/s | 42 kB 00:00 2024-03-08T23:53:23.270 INFO:teuthology.orchestra.run.smithi179.stdout:(155/174): NetworkManager-libnm-1.46.0-1.el9.x8 6.8 MB/s | 1.8 MB 00:00 2024-03-08T23:53:23.295 INFO:teuthology.orchestra.run.smithi179.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 1.6 MB/s | 248 kB 00:00 2024-03-08T23:53:23.320 INFO:teuthology.orchestra.run.smithi179.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-08T23:53:23.346 INFO:teuthology.orchestra.run.smithi179.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 726 kB/s | 36 kB 00:00 2024-03-08T23:53:23.374 INFO:teuthology.orchestra.run.smithi179.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.7 MB/s | 192 kB 00:00 2024-03-08T23:53:23.460 INFO:teuthology.orchestra.run.smithi179.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.2 MB/s | 256 kB 00:00 2024-03-08T23:53:23.488 INFO:teuthology.orchestra.run.smithi179.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 197 kB/s | 23 kB 00:00 2024-03-08T23:53:23.522 INFO:teuthology.orchestra.run.smithi179.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 108 kB 00:00 2024-03-08T23:53:23.547 INFO:teuthology.orchestra.run.smithi179.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-08T23:53:23.572 INFO:teuthology.orchestra.run.smithi179.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 136 kB 00:00 2024-03-08T23:53:23.598 INFO:teuthology.orchestra.run.smithi179.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-08T23:53:23.623 INFO:teuthology.orchestra.run.smithi179.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 572 kB/s | 29 kB 00:00 2024-03-08T23:53:23.815 INFO:teuthology.orchestra.run.smithi179.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 1.2 MB 00:00 2024-03-08T23:53:23.915 INFO:teuthology.orchestra.run.smithi179.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 7.4 MB/s | 2.2 MB 00:00 2024-03-08T23:53:23.990 INFO:teuthology.orchestra.run.smithi179.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 6.1 MB/s | 465 kB 00:00 2024-03-08T23:53:24.041 INFO:teuthology.orchestra.run.smithi179.stdout:(170/174): thrift-0.15.0-2.el9.x86_64.rpm 766 kB/s | 1.6 MB 00:02 2024-03-08T23:53:24.066 INFO:teuthology.orchestra.run.smithi179.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 414 kB/s | 31 kB 00:00 2024-03-08T23:53:24.149 INFO:teuthology.orchestra.run.smithi179.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 894 kB/s | 74 kB 00:00 2024-03-08T23:53:24.233 INFO:teuthology.orchestra.run.smithi179.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.6 MB/s | 1.1 MB 00:00 2024-03-08T23:53:24.300 INFO:teuthology.orchestra.run.smithi179.stdout:(174/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.7 MB/s | 2.3 MB 00:00 2024-03-08T23:53:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:-------------------------------------------------------------------------------- 2024-03-08T23:53:24.304 INFO:teuthology.orchestra.run.smithi179.stdout:Total 11 MB/s | 223 MB 00:19 2024-03-08T23:53:24.359 INFO:teuthology.orchestra.run.smithi073.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:24.361 INFO:teuthology.orchestra.run.smithi073.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:24.375 INFO:teuthology.orchestra.run.smithi073.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T23:53:24.376 INFO:teuthology.orchestra.run.smithi073.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T23:53:24.377 INFO:teuthology.orchestra.run.smithi073.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T23:53:24.378 INFO:teuthology.orchestra.run.smithi073.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:24.380 INFO:teuthology.orchestra.run.smithi073.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:24.383 INFO:teuthology.orchestra.run.smithi073.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T23:53:24.384 INFO:teuthology.orchestra.run.smithi073.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T23:53:24.464 INFO:teuthology.orchestra.run.smithi073.stdout:Dependencies resolved. 2024-03-08T23:53:24.493 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2024-03-08T23:53:24.493 INFO:teuthology.orchestra.run.smithi073.stdout: Package Arch Version Repository Size 2024-03-08T23:53:24.493 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2024-03-08T23:53:24.493 INFO:teuthology.orchestra.run.smithi073.stdout:Installing: 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T23:53:24.494 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T23:53:24.495 INFO:teuthology.orchestra.run.smithi073.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T23:53:24.496 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrading: 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T23:53:24.497 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T23:53:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout:Installing dependencies: 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T23:53:24.499 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T23:53:24.500 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T23:53:24.501 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T23:53:24.502 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T23:53:24.503 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T23:53:24.504 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T23:53:24.505 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T23:53:24.506 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T23:53:24.507 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout:Installing weak dependencies: 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T23:53:24.508 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction Summary 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================================ 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout:Install 150 Packages 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout:Upgrade 24 Packages 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:53:24.509 INFO:teuthology.orchestra.run.smithi073.stdout:Total download size: 223 M 2024-03-08T23:53:24.510 INFO:teuthology.orchestra.run.smithi073.stdout:Downloading Packages: 2024-03-08T23:53:24.765 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction check 2024-03-08T23:53:24.786 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - CRB 3.3 MB/s | 6.1 MB 00:01 2024-03-08T23:53:25.033 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction check succeeded. 2024-03-08T23:53:25.034 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction test 2024-03-08T23:53:25.056 INFO:teuthology.orchestra.run.smithi073.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 38 kB/s | 6.6 kB 00:00 2024-03-08T23:53:25.289 INFO:teuthology.orchestra.run.smithi073.stdout:(2/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 3.7 MB/s | 889 kB 00:00 2024-03-08T23:53:25.339 INFO:teuthology.orchestra.run.smithi073.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 2.8 MB/s | 145 kB 00:00 2024-03-08T23:53:25.606 INFO:teuthology.orchestra.run.smithi073.stdout:(4/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 8.0 MB/s | 2.1 MB 00:00 2024-03-08T23:53:25.798 INFO:teuthology.orchestra.run.smithi073.stdout:(5/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 7.8 MB/s | 1.5 MB 00:00 2024-03-08T23:53:26.145 INFO:teuthology.orchestra.run.smithi102.stdout:CentOS Stream 9 - Extras packages 77 kB/s | 15 kB 00:00 2024-03-08T23:53:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:(6/174): ceph-base-19.0.0-1965.g7d63482a.el9.x8 3.9 MB/s | 5.4 MB 00:01 2024-03-08T23:53:26.382 INFO:teuthology.orchestra.run.smithi073.stdout:(7/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 8.0 MB/s | 4.7 MB 00:00 2024-03-08T23:53:26.813 INFO:teuthology.orchestra.run.smithi179.stdout:Transaction test succeeded. 2024-03-08T23:53:26.813 INFO:teuthology.orchestra.run.smithi179.stdout:Running transaction 2024-03-08T23:53:26.842 INFO:teuthology.orchestra.run.smithi073.stdout:(8/174): ceph-common-19.0.0-1965.g7d63482a.el9. 10 MB/s | 20 MB 00:01 2024-03-08T23:53:26.891 INFO:teuthology.orchestra.run.smithi073.stdout:(9/174): ceph-selinux-19.0.0-1965.g7d63482a.el9 503 kB/s | 25 kB 00:00 2024-03-08T23:53:27.283 INFO:teuthology.orchestra.run.smithi073.stdout:(10/174): ceph-radosgw-19.0.0-1965.g7d63482a.el 11 MB/s | 9.8 MB 00:00 2024-03-08T23:53:27.383 INFO:teuthology.orchestra.run.smithi073.stdout:(11/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 2.2 MB/s | 220 kB 00:00 2024-03-08T23:53:27.450 INFO:teuthology.orchestra.run.smithi073.stdout:(12/174): cephfs-shell-19.0.0-1965.g7d63482a.el 406 kB/s | 27 kB 00:00 2024-03-08T23:53:27.525 INFO:teuthology.orchestra.run.smithi073.stdout:(13/174): libcephfs-devel-19.0.0-1965.g7d63482a 415 kB/s | 31 kB 00:00 2024-03-08T23:53:27.834 INFO:teuthology.orchestra.run.smithi073.stdout:(14/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 2.4 MB/s | 745 kB 00:00 2024-03-08T23:53:27.918 INFO:teuthology.orchestra.run.smithi073.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 1.9 MB/s | 163 kB 00:00 2024-03-08T23:53:27.993 INFO:teuthology.orchestra.run.smithi073.stdout:(16/174): librados-devel-19.0.0-1965.g7d63482a. 1.7 MB/s | 127 kB 00:00 2024-03-08T23:53:28.210 INFO:teuthology.orchestra.run.smithi073.stdout:(17/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x8 8.6 MB/s | 17 MB 00:01 2024-03-08T23:53:28.235 INFO:teuthology.orchestra.run.smithi073.stdout:(18/174): libradosstriper1-19.0.0-1965.g7d63482 2.0 MB/s | 501 kB 00:00 2024-03-08T23:53:28.294 INFO:teuthology.orchestra.run.smithi073.stdout:(19/174): python3-ceph-argparse-19.0.0-1965.g7d 774 kB/s | 45 kB 00:00 2024-03-08T23:53:28.369 INFO:teuthology.orchestra.run.smithi073.stdout:(20/174): python3-ceph-common-19.0.0-1965.g7d63 1.7 MB/s | 131 kB 00:00 2024-03-08T23:53:28.444 INFO:teuthology.orchestra.run.smithi073.stdout:(21/174): python3-cephfs-19.0.0-1965.g7d63482a. 2.1 MB/s | 162 kB 00:00 2024-03-08T23:53:28.536 INFO:teuthology.orchestra.run.smithi073.stdout:(22/174): python3-rados-19.0.0-1965.g7d63482a.e 3.4 MB/s | 320 kB 00:00 2024-03-08T23:53:28.628 INFO:teuthology.orchestra.run.smithi073.stdout:(23/174): python3-rbd-19.0.0-1965.g7d63482a.el9 3.2 MB/s | 298 kB 00:00 2024-03-08T23:53:28.703 INFO:teuthology.orchestra.run.smithi073.stdout:(24/174): python3-rgw-19.0.0-1965.g7d63482a.el9 1.3 MB/s | 99 kB 00:00 2024-03-08T23:53:28.787 INFO:teuthology.orchestra.run.smithi073.stdout:(25/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 8.9 MB/s | 5.1 MB 00:00 2024-03-08T23:53:28.812 INFO:teuthology.orchestra.run.smithi073.stdout:(26/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 785 kB/s | 85 kB 00:00 2024-03-08T23:53:28.896 INFO:teuthology.orchestra.run.smithi073.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 2.0 MB/s | 171 kB 00:00 2024-03-08T23:53:28.954 INFO:teuthology.orchestra.run.smithi073.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1965.g 584 kB/s | 34 kB 00:00 2024-03-08T23:53:29.029 INFO:teuthology.orchestra.run.smithi073.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 1.8 MB/s | 139 kB 00:00 2024-03-08T23:53:29.138 INFO:teuthology.orchestra.run.smithi073.stdout:(30/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 8.8 MB/s | 3.1 MB 00:00 2024-03-08T23:53:29.305 INFO:teuthology.orchestra.run.smithi073.stdout:(31/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 6.5 MB/s | 1.8 MB 00:00 2024-03-08T23:53:29.397 INFO:teuthology.orchestra.run.smithi073.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1965.g7d 2.6 MB/s | 248 kB 00:00 2024-03-08T23:53:29.472 INFO:teuthology.orchestra.run.smithi073.stdout:(33/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 658 kB/s | 49 kB 00:00 2024-03-08T23:53:29.548 INFO:teuthology.orchestra.run.smithi073.stdout:(34/174): ceph-prometheus-alerts-19.0.0-1965.g7 220 kB/s | 16 kB 00:00 2024-03-08T23:53:29.639 INFO:teuthology.orchestra.run.smithi073.stdout:(35/174): ceph-volume-19.0.0-1965.g7d63482a.el9 2.8 MB/s | 264 kB 00:00 2024-03-08T23:53:29.806 INFO:teuthology.orchestra.run.smithi073.stdout:(36/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 4.4 MB/s | 748 kB 00:00 2024-03-08T23:53:29.882 INFO:teuthology.orchestra.run.smithi073.stdout:(37/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 243 kB/s | 18 kB 00:00 2024-03-08T23:53:29.966 INFO:teuthology.orchestra.run.smithi073.stdout:(38/174): ceph-mgr-diskprediction-local-19.0.0- 9.0 MB/s | 7.4 MB 00:00 2024-03-08T23:53:30.377 INFO:teuthology.orchestra.run.smithi073.stdout:(39/174): ceph-test-19.0.0-1965.g7d63482a.el9.x 14 MB/s | 47 MB 00:03 2024-03-08T23:53:30.535 INFO:teuthology.orchestra.run.smithi073.stdout:(40/174): boost-program-options-1.75.0-8.el9.x8 988 kB/s | 106 kB 00:00 2024-03-08T23:53:30.568 INFO:teuthology.orchestra.run.smithi073.stdout:(41/174): automake-1.16.2-8.el9.noarch.rpm 965 kB/s | 662 kB 00:00 2024-03-08T23:53:30.594 INFO:teuthology.orchestra.run.smithi073.stdout:(42/174): bison-3.7.4-5.el9.x86_64.rpm 1.4 MB/s | 920 kB 00:00 2024-03-08T23:53:30.619 INFO:teuthology.orchestra.run.smithi073.stdout:(43/174): flex-2.6.4-9.el9.x86_64.rpm 3.6 MB/s | 307 kB 00:00 2024-03-08T23:53:30.644 INFO:teuthology.orchestra.run.smithi073.stdout:(44/174): flexiblas-3.0.4-8.el9.x86_64.rpm 422 kB/s | 32 kB 00:00 2024-03-08T23:53:30.669 INFO:teuthology.orchestra.run.smithi073.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 343 kB/s | 17 kB 00:00 2024-03-08T23:53:30.753 INFO:teuthology.orchestra.run.smithi073.stdout:(46/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-08T23:53:30.773 INFO:teuthology.orchestra.run.smithi102.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 21 MB 00:04 2024-03-08T23:53:30.795 INFO:teuthology.orchestra.run.smithi073.stdout:(47/174): libaio-devel-0.3.111-13.el9.x86_64.rp 67 kB/s | 10 kB 00:00 2024-03-08T23:53:30.820 INFO:teuthology.orchestra.run.smithi073.stdout:(48/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.0 MB/s | 159 kB 00:00 2024-03-08T23:53:30.845 INFO:teuthology.orchestra.run.smithi073.stdout:(49/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 481 kB/s | 44 kB 00:00 2024-03-08T23:53:30.887 INFO:teuthology.orchestra.run.smithi073.stdout:(50/174): librdkafka-1.6.1-102.el9.x86_64.rpm 9.7 MB/s | 661 kB 00:00 2024-03-08T23:53:30.912 INFO:teuthology.orchestra.run.smithi073.stdout:(51/174): libtool-2.4.6-45.el9.x86_64.rpm 6.2 MB/s | 578 kB 00:00 2024-03-08T23:53:30.938 INFO:teuthology.orchestra.run.smithi073.stdout:(52/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 509 kB/s | 46 kB 00:00 2024-03-08T23:53:30.963 INFO:teuthology.orchestra.run.smithi073.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-08T23:53:30.988 INFO:teuthology.orchestra.run.smithi073.stdout:(54/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-08T23:53:31.014 INFO:teuthology.orchestra.run.smithi073.stdout:(55/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.8 MB/s | 217 kB 00:00 2024-03-08T23:53:31.039 INFO:teuthology.orchestra.run.smithi073.stdout:(56/174): openblas-0.3.21-2.el9.x86_64.rpm 459 kB/s | 34 kB 00:00 2024-03-08T23:53:31.097 INFO:teuthology.orchestra.run.smithi073.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 18 MB/s | 1.5 MB 00:00 2024-03-08T23:53:31.181 INFO:teuthology.orchestra.run.smithi073.stdout:(58/174): openblas-openmp-0.3.21-2.el9.x86_64.r 28 MB/s | 4.7 MB 00:00 2024-03-08T23:53:31.223 INFO:teuthology.orchestra.run.smithi073.stdout:(59/174): postgresql-contrib-13.11-1.el9.x86_64 4.3 MB/s | 811 kB 00:00 2024-03-08T23:53:31.248 INFO:teuthology.orchestra.run.smithi073.stdout:(60/174): postgresql-private-libs-13.11-1.el9.x 906 kB/s | 134 kB 00:00 2024-03-08T23:53:31.340 INFO:teuthology.orchestra.run.smithi073.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 36 MB/s | 5.7 MB 00:00 2024-03-08T23:53:31.357 INFO:teuthology.orchestra.run.smithi073.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 739 kB/s | 78 kB 00:00 2024-03-08T23:53:31.391 INFO:teuthology.orchestra.run.smithi073.stdout:(63/174): protobuf-3.14.0-13.el9.x86_64.rpm 7.3 MB/s | 1.0 MB 00:00 2024-03-08T23:53:31.433 INFO:teuthology.orchestra.run.smithi073.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 336 kB/s | 24 kB 00:00 2024-03-08T23:53:31.499 INFO:teuthology.orchestra.run.smithi073.stdout:(65/174): python3-babel-2.9.1-2.el9.noarch.rpm 36 MB/s | 5.8 MB 00:00 2024-03-08T23:53:31.525 INFO:teuthology.orchestra.run.smithi073.stdout:(66/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-08T23:53:31.550 INFO:teuthology.orchestra.run.smithi073.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.2 MB/s | 144 kB 00:00 2024-03-08T23:53:31.575 INFO:teuthology.orchestra.run.smithi073.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 636 kB/s | 32 kB 00:00 2024-03-08T23:53:31.676 INFO:teuthology.orchestra.run.smithi073.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-08T23:53:31.701 INFO:teuthology.orchestra.run.smithi073.stdout:(70/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.4 MB/s | 217 kB 00:00 2024-03-08T23:53:31.726 INFO:teuthology.orchestra.run.smithi073.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.2 MB/s | 174 kB 00:00 2024-03-08T23:53:31.743 INFO:teuthology.orchestra.run.smithi073.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-08T23:53:31.769 INFO:teuthology.orchestra.run.smithi073.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 989 kB/s | 41 kB 00:00 2024-03-08T23:53:31.794 INFO:teuthology.orchestra.run.smithi073.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-08T23:53:31.819 INFO:teuthology.orchestra.run.smithi073.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.7 MB/s | 132 kB 00:00 2024-03-08T23:53:31.844 INFO:teuthology.orchestra.run.smithi073.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-08T23:53:31.870 INFO:teuthology.orchestra.run.smithi073.stdout:(77/174): python3-pytz-2021.1-5.el9.noarch.rpm 631 kB/s | 47 kB 00:00 2024-03-08T23:53:31.894 INFO:teuthology.orchestra.run.smithi073.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 569 kB/s | 43 kB 00:00 2024-03-08T23:53:31.920 INFO:teuthology.orchestra.run.smithi073.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 776 kB/s | 39 kB 00:00 2024-03-08T23:53:32.070 INFO:teuthology.orchestra.run.smithi073.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 234 kB/s | 41 kB 00:00 2024-03-08T23:53:32.104 INFO:teuthology.orchestra.run.smithi073.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.6 MB/s | 300 kB 00:00 2024-03-08T23:53:32.263 INFO:teuthology.orchestra.run.smithi073.stdout:(82/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-08T23:53:32.288 INFO:teuthology.orchestra.run.smithi073.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 262 kB/s | 56 kB 00:00 2024-03-08T23:53:32.313 INFO:teuthology.orchestra.run.smithi073.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 302 kB/s | 63 kB 00:00 2024-03-08T23:53:32.346 INFO:teuthology.orchestra.run.smithi073.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 758 kB/s | 44 kB 00:00 2024-03-08T23:53:32.397 INFO:teuthology.orchestra.run.smithi073.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 726 kB/s | 79 kB 00:00 2024-03-08T23:53:32.422 INFO:teuthology.orchestra.run.smithi073.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 2.2 MB/s | 241 kB 00:00 2024-03-08T23:53:32.447 INFO:teuthology.orchestra.run.smithi073.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 2.0 MB/s | 103 kB 00:00 2024-03-08T23:53:32.481 INFO:teuthology.orchestra.run.smithi073.stdout:(89/174): python3-cryptography-36.0.1-4.el9.x86 14 MB/s | 1.1 MB 00:00 2024-03-08T23:53:32.506 INFO:teuthology.orchestra.run.smithi073.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.5 MB/s | 124 kB 00:00 2024-03-08T23:53:32.531 INFO:teuthology.orchestra.run.smithi073.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.2 MB/s | 187 kB 00:00 2024-03-08T23:53:32.698 INFO:teuthology.orchestra.run.smithi073.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 978 kB/s | 187 kB 00:00 2024-03-08T23:53:32.723 INFO:teuthology.orchestra.run.smithi073.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.0 MB/s | 195 kB 00:00 2024-03-08T23:53:32.782 INFO:teuthology.orchestra.run.smithi073.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 1.5 MB/s | 125 kB 00:00 2024-03-08T23:53:32.799 INFO:teuthology.orchestra.run.smithi073.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 285 kB/s | 21 kB 00:00 2024-03-08T23:53:32.858 INFO:teuthology.orchestra.run.smithi073.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 338 kB/s | 25 kB 00:00 2024-03-08T23:53:32.891 INFO:teuthology.orchestra.run.smithi073.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-08T23:53:32.983 INFO:teuthology.orchestra.run.smithi073.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 1.9 MB/s | 245 kB 00:00 2024-03-08T23:53:33.026 INFO:teuthology.orchestra.run.smithi073.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 5.1 MB/s | 216 kB 00:00 2024-03-08T23:53:33.059 INFO:teuthology.orchestra.run.smithi073.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.4 MB/s | 248 kB 00:00 2024-03-08T23:53:33.084 INFO:teuthology.orchestra.run.smithi073.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 596 kB/s | 35 kB 00:00 2024-03-08T23:53:33.110 INFO:teuthology.orchestra.run.smithi073.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 883 kB/s | 44 kB 00:00 2024-03-08T23:53:33.135 INFO:teuthology.orchestra.run.smithi073.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-08T23:53:33.160 INFO:teuthology.orchestra.run.smithi073.stdout:(104/174): lua-devel-5.4.4-4.el9.x86_64.rpm 444 kB/s | 22 kB 00:00 2024-03-08T23:53:33.452 INFO:teuthology.orchestra.run.smithi073.stdout:(105/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.7 MB/s | 863 kB 00:00 2024-03-08T23:53:33.594 INFO:teuthology.orchestra.run.smithi073.stdout:(106/174): openssl-devel-3.0.7-27.el9.x86_64.rp 5.2 MB/s | 4.1 MB 00:00 2024-03-08T23:53:33.745 INFO:teuthology.orchestra.run.smithi073.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 939 kB/s | 548 kB 00:00 2024-03-08T23:53:33.861 INFO:teuthology.orchestra.run.smithi073.stdout:(108/174): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-08T23:53:33.874 INFO:teuthology.orchestra.run.smithi179.stdout: Preparing : 1/1 2024-03-08T23:53:33.978 INFO:teuthology.orchestra.run.smithi073.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 588 kB/s | 309 kB 00:00 2024-03-08T23:53:33.995 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T23:53:34.129 INFO:teuthology.orchestra.run.smithi073.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 97 kB/s | 26 kB 00:00 2024-03-08T23:53:34.162 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T23:53:34.294 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:53:34.305 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:53:34.320 INFO:teuthology.orchestra.run.smithi073.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 142 kB/s | 49 kB 00:00 2024-03-08T23:53:34.345 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T23:53:34.403 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T23:53:34.440 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T23:53:34.465 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T23:53:34.471 INFO:teuthology.orchestra.run.smithi073.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 197 kB/s | 67 kB 00:00 2024-03-08T23:53:34.509 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T23:53:34.746 INFO:teuthology.orchestra.run.smithi073.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 355 kB/s | 151 kB 00:00 2024-03-08T23:53:34.813 INFO:teuthology.orchestra.run.smithi073.stdout:(114/174): libarrow-9.0.0-10.el9.x86_64.rpm 4.1 MB/s | 4.4 MB 00:01 2024-03-08T23:53:34.845 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T23:53:34.899 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:53:35.020 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:53:35.049 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:53:35.088 INFO:teuthology.orchestra.run.smithi073.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.3 MB/s | 838 kB 00:00 2024-03-08T23:53:35.140 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:53:35.155 INFO:teuthology.orchestra.run.smithi073.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 127 kB/s | 43 kB 00:00 2024-03-08T23:53:35.170 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:53:35.208 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T23:53:35.347 INFO:teuthology.orchestra.run.smithi073.stdout:(117/174): python3-asyncssh-2.13.2-4.el9.noarch 912 kB/s | 548 kB 00:00 2024-03-08T23:53:35.372 INFO:teuthology.orchestra.run.smithi073.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 113 kB/s | 32 kB 00:00 2024-03-08T23:53:35.401 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T23:53:35.414 INFO:teuthology.orchestra.run.smithi073.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 58 kB/s | 15 kB 00:00 2024-03-08T23:53:35.452 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T23:53:35.480 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:53:35.583 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:53:35.605 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:53:35.653 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:53:35.692 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T23:53:35.716 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:53:35.731 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:53:35.781 INFO:teuthology.orchestra.run.smithi073.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 396 kB/s | 172 kB 00:00 2024-03-08T23:53:35.812 INFO:teuthology.orchestra.run.smithi102.stdout:lab-extras 11 kB/s | 1.7 kB 00:00 2024-03-08T23:53:35.881 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:53:35.913 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:53:35.915 INFO:teuthology.orchestra.run.smithi073.stdout:(121/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 606 kB/s | 303 kB 00:00 2024-03-08T23:53:35.945 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T23:53:35.957 INFO:teuthology.orchestra.run.smithi073.stdout:(122/174): python3-cherrypy-18.6.1-2.el9.noarch 614 kB/s | 358 kB 00:00 2024-03-08T23:53:36.057 INFO:teuthology.orchestra.run.smithi073.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 128 kB/s | 35 kB 00:00 2024-03-08T23:53:36.068 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T23:53:36.181 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:53:36.233 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:53:36.300 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:36.418 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:36.424 INFO:teuthology.orchestra.run.smithi073.stdout:(124/174): python3-google-auth-2.28.0-1.el9.noa 398 kB/s | 203 kB 00:00 2024-03-08T23:53:36.474 INFO:teuthology.orchestra.run.smithi073.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 347 kB/s | 144 kB 00:00 2024-03-08T23:53:36.608 INFO:teuthology.orchestra.run.smithi073.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 58 kB/s | 11 kB 00:00 2024-03-08T23:53:36.637 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:36.741 INFO:teuthology.orchestra.run.smithi073.stdout:(127/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.6 MB/s | 2.0 MB 00:00 2024-03-08T23:53:36.758 INFO:teuthology.orchestra.run.smithi073.stdout:(128/174): python3-jaraco-classes-3.2.1-5.el9.n 62 kB/s | 18 kB 00:00 2024-03-08T23:53:36.759 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T23:53:36.852 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T23:53:36.875 INFO:teuthology.orchestra.run.smithi073.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 87 kB/s | 23 kB 00:00 2024-03-08T23:53:36.889 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T23:53:36.916 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T23:53:36.952 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T23:53:37.017 INFO:teuthology.orchestra.run.smithi073.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 71 kB/s | 19 kB 00:00 2024-03-08T23:53:37.026 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T23:53:37.034 INFO:teuthology.orchestra.run.smithi073.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 71 kB/s | 20 kB 00:00 2024-03-08T23:53:37.080 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T23:53:37.118 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T23:53:37.183 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T23:53:37.217 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T23:53:37.260 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T23:53:37.293 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T23:53:37.321 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T23:53:37.351 INFO:teuthology.orchestra.run.smithi073.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 139 kB/s | 46 kB 00:00 2024-03-08T23:53:37.376 INFO:teuthology.orchestra.run.smithi073.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 231 kB/s | 79 kB 00:00 2024-03-08T23:53:37.390 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T23:53:37.426 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T23:53:37.469 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T23:53:37.506 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T23:53:37.543 INFO:teuthology.orchestra.run.smithi073.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 1.5 MB/s | 1.0 MB 00:00 2024-03-08T23:53:37.577 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T23:53:37.621 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T23:53:37.702 INFO:teuthology.orchestra.run.smithi073.stdout:(135/174): python3-natsort-7.1.1-5.el9.noarch.r 164 kB/s | 58 kB 00:00 2024-03-08T23:53:37.703 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T23:53:37.819 INFO:teuthology.orchestra.run.smithi073.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 60 kB/s | 16 kB 00:00 2024-03-08T23:53:37.834 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T23:53:37.877 INFO:teuthology.orchestra.run.smithi073.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 544 kB/s | 272 kB 00:00 2024-03-08T23:53:37.890 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T23:53:38.053 INFO:teuthology.orchestra.run.smithi073.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 257 kB/s | 90 kB 00:00 2024-03-08T23:53:38.087 INFO:teuthology.orchestra.run.smithi073.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 106 kB/s | 28 kB 00:00 2024-03-08T23:53:38.145 INFO:teuthology.orchestra.run.smithi073.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 115 kB/s | 31 kB 00:00 2024-03-08T23:53:38.259 INFO:teuthology.orchestra.run.smithi102.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:38.260 INFO:teuthology.orchestra.run.smithi102.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-08T23:53:38.274 INFO:teuthology.orchestra.run.smithi102.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-08T23:53:38.276 INFO:teuthology.orchestra.run.smithi102.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-08T23:53:38.277 INFO:teuthology.orchestra.run.smithi102.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-08T23:53:38.278 INFO:teuthology.orchestra.run.smithi102.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:38.280 INFO:teuthology.orchestra.run.smithi102.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-08T23:53:38.286 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-08T23:53:38.286 INFO:teuthology.orchestra.run.smithi102.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-08T23:53:38.363 INFO:teuthology.orchestra.run.smithi102.stdout:Dependencies resolved. 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout: Package Arch Version Repository Size 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout:Installing: 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-08T23:53:38.395 INFO:teuthology.orchestra.run.smithi102.stdout: ceph x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 6.6 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.4 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 889 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 145 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 1.5 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 139 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 1.8 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 7.4 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 49 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 9.8 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 47 M 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 264 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 748 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 220 k 2024-03-08T23:53:38.396 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 27 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 18 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 31 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 745 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 127 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-08T23:53:38.397 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 162 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 320 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 298 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 99 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 85 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 171 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrading: 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager x86_64 1:1.46.0-1.el9 baseos 2.3 M 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm x86_64 1:1.46.0-1.el9 baseos 1.8 M 2024-03-08T23:53:38.398 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team x86_64 1:1.46.0-1.el9 baseos 42 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui x86_64 1:1.46.0-1.el9 baseos 248 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: librados2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.4 M 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 3.1 M 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-08T23:53:38.399 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout:Installing dependencies: 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-08T23:53:38.400 INFO:teuthology.orchestra.run.smithi102.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 20 M 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 34 k 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 2.1 M 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 248 k 2024-03-08T23:53:38.401 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 4.7 M 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 17 M 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts noarch 2:19.0.0-1965.g7d63482a.el9 ceph-noarch 16 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 25 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-08T23:53:38.402 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 163 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 501 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2 x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 5.1 M 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-08T23:53:38.403 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh noarch 2.13.2-4.el9 epel 548 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-08T23:53:38.404 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 45 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common x86_64 2:19.0.0-1965.g7d63482a.el9 ceph 131 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-08T23:53:38.405 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth noarch 1:2.28.0-1.el9 epel 203 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-08T23:53:38.406 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-08T23:53:38.407 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-08T23:53:38.408 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-08T23:53:38.409 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:Installing weak dependencies: 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction Summary 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:============================================================================================ 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:Install 150 Packages 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout:Upgrade 24 Packages 2024-03-08T23:53:38.410 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:53:38.411 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T23:53:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:Total download size: 223 M 2024-03-08T23:53:38.411 INFO:teuthology.orchestra.run.smithi102.stdout:Downloading Packages: 2024-03-08T23:53:38.412 INFO:teuthology.orchestra.run.smithi073.stdout:(141/174): python3-tempora-5.0.0-2.el9.noarch.r 134 kB/s | 36 kB 00:00 2024-03-08T23:53:38.437 INFO:teuthology.orchestra.run.smithi073.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 169 kB/s | 59 kB 00:00 2024-03-08T23:53:38.485 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T23:53:38.558 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T23:53:38.563 INFO:teuthology.orchestra.run.smithi073.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 370 kB/s | 188 kB 00:00 2024-03-08T23:53:38.605 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:53:38.612 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:53:38.645 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T23:53:38.694 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T23:53:38.732 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T23:53:38.754 INFO:teuthology.orchestra.run.smithi073.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 149 kB/s | 51 kB 00:00 2024-03-08T23:53:38.761 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T23:53:38.946 INFO:teuthology.orchestra.run.smithi073.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 451 kB/s | 230 kB 00:00 2024-03-08T23:53:38.980 INFO:teuthology.orchestra.run.smithi073.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 215 kB/s | 90 kB 00:00 2024-03-08T23:53:38.987 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T23:53:39.149 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T23:53:39.222 INFO:teuthology.orchestra.run.smithi073.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 73 kB/s | 20 kB 00:00 2024-03-08T23:53:39.239 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T23:53:39.283 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T23:53:39.364 INFO:teuthology.orchestra.run.smithi073.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 701 kB/s | 427 kB 00:00 2024-03-08T23:53:39.410 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T23:53:39.447 INFO:teuthology.orchestra.run.smithi102.stdout:(1/174): ceph-19.0.0-1965.g7d63482a.el9.x86_64. 28 kB/s | 6.6 kB 00:00 2024-03-08T23:53:39.453 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T23:53:39.489 INFO:teuthology.orchestra.run.smithi073.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 377 kB/s | 192 kB 00:00 2024-03-08T23:53:39.525 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T23:53:39.597 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T23:53:39.629 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T23:53:39.665 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T23:53:39.712 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T23:53:39.748 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T23:53:39.800 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T23:53:39.842 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T23:53:39.915 INFO:teuthology.orchestra.run.smithi073.stdout:(150/174): librados2-19.0.0-1965.g7d63482a.el9. 6.1 MB/s | 3.4 MB 00:00 2024-03-08T23:53:39.956 INFO:teuthology.orchestra.run.smithi073.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-08T23:53:39.976 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T23:53:39.981 INFO:teuthology.orchestra.run.smithi102.stdout:(2/174): ceph-base-19.0.0-1965.g7d63482a.el9.x8 6.9 MB/s | 5.4 MB 00:00 2024-03-08T23:53:39.998 INFO:teuthology.orchestra.run.smithi073.stdout:(152/174): thrift-0.15.0-2.el9.x86_64.rpm 2.0 MB/s | 1.6 MB 00:00 2024-03-08T23:53:40.014 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T23:53:40.045 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T23:53:40.048 INFO:teuthology.orchestra.run.smithi102.stdout:(3/174): ceph-immutable-object-cache-19.0.0-196 2.1 MB/s | 145 kB 00:00 2024-03-08T23:53:40.086 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T23:53:40.180 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T23:53:40.220 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T23:53:40.260 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T23:53:40.273 INFO:teuthology.orchestra.run.smithi102.stdout:(4/174): ceph-mds-19.0.0-1965.g7d63482a.el9.x86 9.5 MB/s | 2.1 MB 00:00 2024-03-08T23:53:40.285 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T23:53:40.310 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T23:53:40.382 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T23:53:40.415 INFO:teuthology.orchestra.run.smithi073.stdout:(153/174): NetworkManager-1.46.0-1.el9.x86_64.r 5.0 MB/s | 2.3 MB 00:00 2024-03-08T23:53:40.448 INFO:teuthology.orchestra.run.smithi102.stdout:(5/174): ceph-mgr-19.0.0-1965.g7d63482a.el9.x86 8.5 MB/s | 1.5 MB 00:00 2024-03-08T23:53:40.457 INFO:teuthology.orchestra.run.smithi073.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 4.0 MB/s | 1.8 MB 00:00 2024-03-08T23:53:40.474 INFO:teuthology.orchestra.run.smithi102.stdout:(6/174): ceph-fuse-19.0.0-1965.g7d63482a.el9.x8 866 kB/s | 889 kB 00:01 2024-03-08T23:53:40.483 INFO:teuthology.orchestra.run.smithi073.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 633 kB/s | 42 kB 00:00 2024-03-08T23:53:40.509 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T23:53:40.534 INFO:teuthology.orchestra.run.smithi073.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-08T23:53:40.558 INFO:teuthology.orchestra.run.smithi073.stdout:(157/174): NetworkManager-tui-1.46.0-1.el9.x86_ 2.4 MB/s | 248 kB 00:00 2024-03-08T23:53:40.575 INFO:teuthology.orchestra.run.smithi073.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 870 kB/s | 36 kB 00:00 2024-03-08T23:53:40.610 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T23:53:40.617 INFO:teuthology.orchestra.run.smithi073.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-08T23:53:40.643 INFO:teuthology.orchestra.run.smithi073.stdout:(160/174): elfutils-libs-0.190-2.el9.x86_64.rpm 3.7 MB/s | 256 kB 00:00 2024-03-08T23:53:40.678 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T23:53:40.678 INFO:teuthology.orchestra.run.smithi073.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 458 kB/s | 23 kB 00:00 2024-03-08T23:53:40.693 INFO:teuthology.orchestra.run.smithi073.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-08T23:53:40.700 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T23:53:40.726 INFO:teuthology.orchestra.run.smithi073.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-08T23:53:40.755 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T23:53:40.760 INFO:teuthology.orchestra.run.smithi073.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.0 MB/s | 136 kB 00:00 2024-03-08T23:53:40.785 INFO:teuthology.orchestra.run.smithi073.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.1 MB/s | 64 kB 00:00 2024-03-08T23:53:40.811 INFO:teuthology.orchestra.run.smithi073.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 573 kB/s | 29 kB 00:00 2024-03-08T23:53:40.882 INFO:teuthology.orchestra.run.smithi102.stdout:(7/174): ceph-mon-19.0.0-1965.g7d63482a.el9.x86 11 MB/s | 4.7 MB 00:00 2024-03-08T23:53:40.939 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T23:53:40.961 INFO:teuthology.orchestra.run.smithi073.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 6.9 MB/s | 1.2 MB 00:00 2024-03-08T23:53:40.980 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T23:53:41.061 INFO:teuthology.orchestra.run.smithi073.stdout:(168/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 8.6 MB/s | 2.2 MB 00:00 2024-03-08T23:53:41.109 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T23:53:41.153 INFO:teuthology.orchestra.run.smithi073.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 5.0 MB/s | 465 kB 00:00 2024-03-08T23:53:41.154 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T23:53:41.183 INFO:teuthology.orchestra.run.smithi102.stdout:(8/174): ceph-common-19.0.0-1965.g7d63482a.el9. 10 MB/s | 20 MB 00:01 2024-03-08T23:53:41.186 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T23:53:41.212 INFO:teuthology.orchestra.run.smithi073.stdout:(170/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 1.8 MB/s | 3.1 MB 00:01 2024-03-08T23:53:41.217 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T23:53:41.233 INFO:teuthology.orchestra.run.smithi102.stdout:(9/174): ceph-selinux-19.0.0-1965.g7d63482a.el9 506 kB/s | 25 kB 00:00 2024-03-08T23:53:41.237 INFO:teuthology.orchestra.run.smithi073.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 373 kB/s | 31 kB 00:00 2024-03-08T23:53:41.245 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T23:53:41.279 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T23:53:41.287 INFO:teuthology.orchestra.run.smithi073.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 7.0 MB/s | 2.3 MB 00:00 2024-03-08T23:53:41.318 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T23:53:41.350 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T23:53:41.363 INFO:teuthology.orchestra.run.smithi073.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 595 kB/s | 74 kB 00:00 2024-03-08T23:53:41.380 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T23:53:41.455 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T23:53:41.498 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T23:53:41.532 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T23:53:41.573 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T23:53:41.596 INFO:teuthology.orchestra.run.smithi073.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 2.8 MB/s | 1.1 MB 00:00 2024-03-08T23:53:41.600 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-03-08T23:53:41.600 INFO:teuthology.orchestra.run.smithi073.stdout:Total 13 MB/s | 223 MB 00:17 2024-03-08T23:53:41.608 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T23:53:41.683 INFO:teuthology.orchestra.run.smithi102.stdout:(10/174): ceph-radosgw-19.0.0-1965.g7d63482a.el 12 MB/s | 9.8 MB 00:00 2024-03-08T23:53:41.696 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T23:53:41.742 INFO:teuthology.orchestra.run.smithi102.stdout:(11/174): cephfs-mirror-19.0.0-1965.g7d63482a.e 3.7 MB/s | 220 kB 00:00 2024-03-08T23:53:41.778 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T23:53:41.792 INFO:teuthology.orchestra.run.smithi102.stdout:(12/174): cephfs-shell-19.0.0-1965.g7d63482a.el 541 kB/s | 27 kB 00:00 2024-03-08T23:53:41.823 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T23:53:41.834 INFO:teuthology.orchestra.run.smithi102.stdout:(13/174): libcephfs-devel-19.0.0-1965.g7d63482a 749 kB/s | 31 kB 00:00 2024-03-08T23:53:41.853 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T23:53:41.892 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T23:53:41.951 INFO:teuthology.orchestra.run.smithi102.stdout:(14/174): libcephfs2-19.0.0-1965.g7d63482a.el9. 6.2 MB/s | 745 kB 00:00 2024-03-08T23:53:41.989 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T23:53:42.018 INFO:teuthology.orchestra.run.smithi102.stdout:(15/174): libcephsqlite-19.0.0-1965.g7d63482a.e 2.4 MB/s | 163 kB 00:00 2024-03-08T23:53:42.038 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T23:53:42.047 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction check 2024-03-08T23:53:42.085 INFO:teuthology.orchestra.run.smithi102.stdout:(16/174): librados-devel-19.0.0-1965.g7d63482a. 1.9 MB/s | 127 kB 00:00 2024-03-08T23:53:42.171 INFO:teuthology.orchestra.run.smithi102.stdout:(17/174): libradosstriper1-19.0.0-1965.g7d63482 5.9 MB/s | 501 kB 00:00 2024-03-08T23:53:42.358 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction check succeeded. 2024-03-08T23:53:42.358 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction test 2024-03-08T23:53:42.636 INFO:teuthology.orchestra.run.smithi102.stdout:(18/174): librgw2-19.0.0-1965.g7d63482a.el9.x86 11 MB/s | 5.1 MB 00:00 2024-03-08T23:53:42.702 INFO:teuthology.orchestra.run.smithi102.stdout:(19/174): python3-ceph-argparse-19.0.0-1965.g7d 679 kB/s | 45 kB 00:00 2024-03-08T23:53:42.766 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T23:53:42.769 INFO:teuthology.orchestra.run.smithi102.stdout:(20/174): python3-ceph-common-19.0.0-1965.g7d63 1.9 MB/s | 131 kB 00:00 2024-03-08T23:53:42.819 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T23:53:42.836 INFO:teuthology.orchestra.run.smithi102.stdout:(21/174): python3-cephfs-19.0.0-1965.g7d63482a. 2.4 MB/s | 162 kB 00:00 2024-03-08T23:53:42.853 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T23:53:42.901 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T23:53:42.911 INFO:teuthology.orchestra.run.smithi102.stdout:(22/174): python3-rados-19.0.0-1965.g7d63482a.e 4.2 MB/s | 320 kB 00:00 2024-03-08T23:53:42.987 INFO:teuthology.orchestra.run.smithi102.stdout:(23/174): python3-rbd-19.0.0-1965.g7d63482a.el9 3.9 MB/s | 298 kB 00:00 2024-03-08T23:53:42.988 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T23:53:43.020 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T23:53:43.054 INFO:teuthology.orchestra.run.smithi102.stdout:(24/174): python3-rgw-19.0.0-1965.g7d63482a.el9 1.5 MB/s | 99 kB 00:00 2024-03-08T23:53:43.113 INFO:teuthology.orchestra.run.smithi102.stdout:(25/174): rbd-fuse-19.0.0-1965.g7d63482a.el9.x8 1.4 MB/s | 85 kB 00:00 2024-03-08T23:53:43.199 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T23:53:43.226 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T23:53:43.304 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T23:53:43.421 INFO:teuthology.orchestra.run.smithi102.stdout:(26/174): rbd-mirror-19.0.0-1965.g7d63482a.el9. 10 MB/s | 3.1 MB 00:00 2024-03-08T23:53:43.480 INFO:teuthology.orchestra.run.smithi102.stdout:(27/174): rbd-nbd-19.0.0-1965.g7d63482a.el9.x86 2.9 MB/s | 171 kB 00:00 2024-03-08T23:53:43.538 INFO:teuthology.orchestra.run.smithi102.stdout:(28/174): ceph-grafana-dashboards-19.0.0-1965.g 581 kB/s | 34 kB 00:00 2024-03-08T23:53:43.597 INFO:teuthology.orchestra.run.smithi102.stdout:(29/174): ceph-mgr-cephadm-19.0.0-1965.g7d63482 2.3 MB/s | 139 kB 00:00 2024-03-08T23:53:43.789 INFO:teuthology.orchestra.run.smithi102.stdout:(30/174): ceph-mgr-dashboard-19.0.0-1965.g7d634 9.4 MB/s | 1.8 MB 00:00 2024-03-08T23:53:43.930 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T23:53:44.040 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T23:53:44.366 INFO:teuthology.orchestra.run.smithi073.stdout:Transaction test succeeded. 2024-03-08T23:53:44.367 INFO:teuthology.orchestra.run.smithi073.stdout:Running transaction 2024-03-08T23:53:44.381 INFO:teuthology.orchestra.run.smithi102.stdout:(31/174): ceph-test-19.0.0-1965.g7d63482a.el9.x 15 MB/s | 47 MB 00:03 2024-03-08T23:53:44.461 INFO:teuthology.orchestra.run.smithi102.stdout:(32/174): ceph-mgr-modules-core-19.0.0-1965.g7d 3.0 MB/s | 248 kB 00:00 2024-03-08T23:53:44.565 INFO:teuthology.orchestra.run.smithi102.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 9.6 MB/s | 7.4 MB 00:00 2024-03-08T23:53:44.582 INFO:teuthology.orchestra.run.smithi102.stdout:(34/174): ceph-mgr-rook-19.0.0-1965.g7d63482a.e 411 kB/s | 49 kB 00:00 2024-03-08T23:53:44.599 INFO:teuthology.orchestra.run.smithi102.stdout:(35/174): ceph-prometheus-alerts-19.0.0-1965.g7 494 kB/s | 16 kB 00:00 2024-03-08T23:53:44.641 INFO:teuthology.orchestra.run.smithi102.stdout:(36/174): ceph-volume-19.0.0-1965.g7d63482a.el9 4.4 MB/s | 264 kB 00:00 2024-03-08T23:53:44.700 INFO:teuthology.orchestra.run.smithi102.stdout:(37/174): cephadm-19.0.0-1965.g7d63482a.el9.noa 7.3 MB/s | 748 kB 00:00 2024-03-08T23:53:44.716 INFO:teuthology.orchestra.run.smithi102.stdout:(38/174): cephfs-top-19.0.0-1965.g7d63482a.el9. 243 kB/s | 18 kB 00:00 2024-03-08T23:53:44.867 INFO:teuthology.orchestra.run.smithi102.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-08T23:53:44.900 INFO:teuthology.orchestra.run.smithi102.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.9 MB/s | 920 kB 00:00 2024-03-08T23:53:44.926 INFO:teuthology.orchestra.run.smithi102.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.8 MB/s | 106 kB 00:00 2024-03-08T23:53:44.959 INFO:teuthology.orchestra.run.smithi102.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.2 MB/s | 307 kB 00:00 2024-03-08T23:53:44.976 INFO:teuthology.orchestra.run.smithi102.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 638 kB/s | 32 kB 00:00 2024-03-08T23:53:45.051 INFO:teuthology.orchestra.run.smithi102.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 33 MB/s | 3.0 MB 00:00 2024-03-08T23:53:45.077 INFO:teuthology.orchestra.run.smithi102.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 172 kB/s | 17 kB 00:00 2024-03-08T23:53:45.102 INFO:teuthology.orchestra.run.smithi102.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 200 kB/s | 10 kB 00:00 2024-03-08T23:53:45.127 INFO:teuthology.orchestra.run.smithi102.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 3.1 MB/s | 159 kB 00:00 2024-03-08T23:53:45.153 INFO:teuthology.orchestra.run.smithi102.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 881 kB/s | 44 kB 00:00 2024-03-08T23:53:45.186 INFO:teuthology.orchestra.run.smithi102.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-08T23:53:45.220 INFO:teuthology.orchestra.run.smithi102.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.5 MB/s | 578 kB 00:00 2024-03-08T23:53:45.243 INFO:teuthology.orchestra.run.smithi102.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 840 kB/s | 46 kB 00:00 2024-03-08T23:53:45.270 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T23:53:45.285 INFO:teuthology.orchestra.run.smithi102.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.3 MB/s | 282 kB 00:00 2024-03-08T23:53:45.310 INFO:teuthology.orchestra.run.smithi102.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 2.7 MB/s | 187 kB 00:00 2024-03-08T23:53:45.336 INFO:teuthology.orchestra.run.smithi102.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-08T23:53:45.343 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T23:53:45.361 INFO:teuthology.orchestra.run.smithi102.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 682 kB/s | 34 kB 00:00 2024-03-08T23:53:45.374 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T23:53:45.405 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T23:53:45.433 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T23:53:45.453 INFO:teuthology.orchestra.run.smithi102.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 41 MB/s | 4.7 MB 00:00 2024-03-08T23:53:45.495 INFO:teuthology.orchestra.run.smithi102.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 12 MB/s | 1.5 MB 00:00 2024-03-08T23:53:45.528 INFO:teuthology.orchestra.run.smithi102.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-08T23:53:45.554 INFO:teuthology.orchestra.run.smithi102.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-08T23:53:45.671 INFO:teuthology.orchestra.run.smithi102.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 40 MB/s | 5.7 MB 00:00 2024-03-08T23:53:45.704 INFO:teuthology.orchestra.run.smithi102.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 6.8 MB/s | 1.0 MB 00:00 2024-03-08T23:53:45.730 INFO:teuthology.orchestra.run.smithi102.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-08T23:53:45.746 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T23:53:45.771 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:53:45.838 INFO:teuthology.orchestra.run.smithi102.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 43 MB/s | 5.8 MB 00:00 2024-03-08T23:53:45.864 INFO:teuthology.orchestra.run.smithi102.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 186 kB/s | 24 kB 00:00 2024-03-08T23:53:45.870 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:53:45.889 INFO:teuthology.orchestra.run.smithi102.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 361 kB/s | 18 kB 00:00 2024-03-08T23:53:45.901 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T23:53:45.914 INFO:teuthology.orchestra.run.smithi102.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 2.8 MB/s | 144 kB 00:00 2024-03-08T23:53:45.937 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T23:53:45.939 INFO:teuthology.orchestra.run.smithi102.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 631 kB/s | 32 kB 00:00 2024-03-08T23:53:46.040 INFO:teuthology.orchestra.run.smithi102.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 38 MB/s | 4.8 MB 00:00 2024-03-08T23:53:46.066 INFO:teuthology.orchestra.run.smithi102.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.7 MB/s | 217 kB 00:00 2024-03-08T23:53:46.091 INFO:teuthology.orchestra.run.smithi102.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 3.4 MB/s | 174 kB 00:00 2024-03-08T23:53:46.116 INFO:teuthology.orchestra.run.smithi102.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-08T23:53:46.133 INFO:teuthology.orchestra.run.smithi102.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 982 kB/s | 41 kB 00:00 2024-03-08T23:53:46.158 INFO:teuthology.orchestra.run.smithi102.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 5.5 MB/s | 238 kB 00:00 2024-03-08T23:53:46.184 INFO:teuthology.orchestra.run.smithi102.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.6 MB/s | 132 kB 00:00 2024-03-08T23:53:46.209 INFO:teuthology.orchestra.run.smithi102.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 4.1 MB/s | 211 kB 00:00 2024-03-08T23:53:46.226 INFO:teuthology.orchestra.run.smithi102.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-08T23:53:46.243 INFO:teuthology.orchestra.run.smithi102.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.3 MB/s | 43 kB 00:00 2024-03-08T23:53:46.310 INFO:teuthology.orchestra.run.smithi102.stdout:(78/174): python3-toml-0.10.2-6.el9.noarch.rpm 592 kB/s | 39 kB 00:00 2024-03-08T23:53:46.382 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T23:53:46.407 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:53:46.461 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:53:46.484 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T23:53:46.544 INFO:teuthology.orchestra.run.smithi102.stdout:(79/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-08T23:53:46.560 INFO:teuthology.orchestra.run.smithi102.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 165 kB/s | 41 kB 00:00 2024-03-08T23:53:46.586 INFO:teuthology.orchestra.run.smithi102.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-08T23:53:46.602 INFO:teuthology.orchestra.run.smithi102.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 1.3 MB/s | 56 kB 00:00 2024-03-08T23:53:46.619 INFO:teuthology.orchestra.run.smithi102.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-08T23:53:46.636 INFO:teuthology.orchestra.run.smithi102.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-08T23:53:46.662 INFO:teuthology.orchestra.run.smithi102.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.8 MB/s | 79 kB 00:00 2024-03-08T23:53:46.687 INFO:teuthology.orchestra.run.smithi102.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-08T23:53:46.721 INFO:teuthology.orchestra.run.smithi102.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 20 MB/s | 1.1 MB 00:00 2024-03-08T23:53:46.746 INFO:teuthology.orchestra.run.smithi102.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.8 MB/s | 103 kB 00:00 2024-03-08T23:53:46.771 INFO:teuthology.orchestra.run.smithi102.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.4 MB/s | 124 kB 00:00 2024-03-08T23:53:46.797 INFO:teuthology.orchestra.run.smithi102.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-08T23:53:46.939 INFO:teuthology.orchestra.run.smithi102.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-08T23:53:47.023 INFO:teuthology.orchestra.run.smithi102.stdout:(92/174): libquadmath-11.4.1-3.el9.x86_64.rpm 2.3 MB/s | 195 kB 00:00 2024-03-08T23:53:47.048 INFO:teuthology.orchestra.run.smithi102.stdout:(93/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.1 MB/s | 803 kB 00:00 2024-03-08T23:53:47.073 INFO:teuthology.orchestra.run.smithi102.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.4 MB/s | 125 kB 00:00 2024-03-08T23:53:47.098 INFO:teuthology.orchestra.run.smithi102.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 428 kB/s | 21 kB 00:00 2024-03-08T23:53:47.123 INFO:teuthology.orchestra.run.smithi102.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 508 kB/s | 25 kB 00:00 2024-03-08T23:53:47.208 INFO:teuthology.orchestra.run.smithi102.stdout:(97/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.9 MB/s | 245 kB 00:00 2024-03-08T23:53:47.433 INFO:teuthology.orchestra.run.smithi102.stdout:(98/174): python3-jinja2-2.11.3-5.el9.noarch.rp 1.1 MB/s | 248 kB 00:00 2024-03-08T23:53:47.516 INFO:teuthology.orchestra.run.smithi102.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 2.5 MB/s | 216 kB 00:00 2024-03-08T23:53:47.568 INFO:teuthology.orchestra.run.smithi102.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 678 kB/s | 35 kB 00:00 2024-03-08T23:53:47.618 INFO:teuthology.orchestra.run.smithi102.stdout:(101/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 887 kB/s | 44 kB 00:00 2024-03-08T23:53:47.669 INFO:teuthology.orchestra.run.smithi102.stdout:(102/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.8 MB/s | 94 kB 00:00 2024-03-08T23:53:47.761 INFO:teuthology.orchestra.run.smithi102.stdout:(103/174): lua-devel-5.4.4-4.el9.x86_64.rpm 243 kB/s | 22 kB 00:00 2024-03-08T23:53:47.894 INFO:teuthology.orchestra.run.smithi102.stdout:(104/174): protobuf-compiler-3.14.0-13.el9.x86_ 6.3 MB/s | 863 kB 00:00 2024-03-08T23:53:48.145 INFO:teuthology.orchestra.run.smithi102.stdout:(105/174): openssl-devel-3.0.7-27.el9.x86_64.rp 3.9 MB/s | 4.1 MB 00:01 2024-03-08T23:53:48.370 INFO:teuthology.orchestra.run.smithi102.stdout:(106/174): ceph-osd-19.0.0-1965.g7d63482a.el9.x 2.1 MB/s | 17 MB 00:07 2024-03-08T23:53:48.688 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:53:48.722 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:53:48.837 INFO:teuthology.orchestra.run.smithi102.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 44 kB/s | 19 kB 00:00 2024-03-08T23:53:49.104 INFO:teuthology.orchestra.run.smithi102.stdout:(108/174): abseil-cpp-20211102.0-3.el9.x86_64.r 453 kB/s | 548 kB 00:01 2024-03-08T23:53:49.162 INFO:teuthology.orchestra.run.smithi102.stdout:(109/174): gperftools-libs-2.9.1-2.el9.x86_64.r 303 kB/s | 309 kB 00:01 2024-03-08T23:53:49.196 INFO:teuthology.orchestra.run.smithi102.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 282 kB/s | 26 kB 00:00 2024-03-08T23:53:49.255 INFO:teuthology.orchestra.run.smithi102.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 528 kB/s | 49 kB 00:00 2024-03-08T23:53:49.259 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:53:49.296 INFO:teuthology.orchestra.run.smithi102.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 671 kB/s | 67 kB 00:00 2024-03-08T23:53:49.309 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:53:49.373 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:53:49.373 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T23:53:49.373 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T23:53:49.373 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:53:49.402 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:53:49.522 INFO:teuthology.orchestra.run.smithi102.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-03-08T23:53:50.264 INFO:teuthology.orchestra.run.smithi102.stdout:(114/174): python3-asyncssh-2.13.2-4.el9.noarch 738 kB/s | 548 kB 00:00 2024-03-08T23:53:50.381 INFO:teuthology.orchestra.run.smithi102.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 773 kB/s | 838 kB 00:01 2024-03-08T23:53:50.397 INFO:teuthology.orchestra.run.smithi102.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 325 kB/s | 43 kB 00:00 2024-03-08T23:53:50.481 INFO:teuthology.orchestra.run.smithi102.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 322 kB/s | 32 kB 00:00 2024-03-08T23:53:50.498 INFO:teuthology.orchestra.run.smithi102.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 149 kB/s | 15 kB 00:00 2024-03-08T23:53:50.748 INFO:teuthology.orchestra.run.smithi102.stdout:(119/174): python3-cheroot-8.6.0-4.el9.noarch.r 643 kB/s | 172 kB 00:00 2024-03-08T23:53:51.015 INFO:teuthology.orchestra.run.smithi102.stdout:(120/174): python3-cherrypy-18.6.1-2.el9.noarch 693 kB/s | 358 kB 00:00 2024-03-08T23:53:51.116 INFO:teuthology.orchestra.run.smithi102.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 354 kB/s | 35 kB 00:00 2024-03-08T23:53:51.199 INFO:teuthology.orchestra.run.smithi102.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 673 kB/s | 303 kB 00:00 2024-03-08T23:53:51.383 INFO:teuthology.orchestra.run.smithi102.stdout:(123/174): python3-google-auth-2.28.0-1.el9.noa 759 kB/s | 203 kB 00:00 2024-03-08T23:53:51.438 INFO:teuthology.orchestra.run.smithi073.stdout: Preparing : 1/1 2024-03-08T23:53:51.551 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T23:53:51.586 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T23:53:51.632 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:53:51.641 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:53:51.650 INFO:teuthology.orchestra.run.smithi102.stdout:(124/174): python3-grpcio-tools-1.46.7-10.el9.x 541 kB/s | 144 kB 00:00 2024-03-08T23:53:51.690 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T23:53:51.744 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T23:53:51.750 INFO:teuthology.orchestra.run.smithi102.stdout:(125/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-08T23:53:51.777 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T23:53:51.803 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T23:53:51.862 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T23:53:51.867 INFO:teuthology.orchestra.run.smithi102.stdout:(126/174): python3-jaraco-classes-3.2.1-5.el9.n 151 kB/s | 18 kB 00:00 2024-03-08T23:53:52.001 INFO:teuthology.orchestra.run.smithi102.stdout:(127/174): python3-jaraco-collections-3.0.0-8.e 174 kB/s | 23 kB 00:00 2024-03-08T23:53:52.101 INFO:teuthology.orchestra.run.smithi102.stdout:(128/174): python3-jaraco-functools-3.5.0-2.el9 194 kB/s | 19 kB 00:00 2024-03-08T23:53:52.188 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T23:53:52.201 INFO:teuthology.orchestra.run.smithi102.stdout:(129/174): python3-jaraco-text-3.2.0-6.el9.noar 197 kB/s | 20 kB 00:00 2024-03-08T23:53:52.249 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:53:52.371 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:53:52.401 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:53:52.492 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:53:52.526 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:53:52.571 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T23:53:52.748 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T23:53:52.803 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T23:53:52.831 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:53:52.934 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:53:52.957 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:53:53.022 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:53:53.069 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T23:53:53.092 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:53:53.106 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:53:53.264 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:53:53.298 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:53:53.330 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T23:53:53.525 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T23:53:53.585 INFO:teuthology.orchestra.run.smithi102.stdout:(130/174): python3-kubernetes-26.1.0-2.el9.noar 756 kB/s | 1.0 MB 00:01 2024-03-08T23:53:53.652 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:53:53.685 INFO:teuthology.orchestra.run.smithi102.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 463 kB/s | 46 kB 00:00 2024-03-08T23:53:53.696 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:53:53.794 INFO:teuthology.orchestra.run.smithi102.stdout:(132/174): python3-more-itertools-8.12.0-2.el9. 728 kB/s | 79 kB 00:00 2024-03-08T23:53:53.801 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:53.836 INFO:teuthology.orchestra.run.smithi102.stdout:(133/174): python3-grpcio-1.46.7-10.el9.x86_64. 791 kB/s | 2.0 MB 00:02 2024-03-08T23:53:53.856 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:53.895 INFO:teuthology.orchestra.run.smithi102.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 575 kB/s | 58 kB 00:00 2024-03-08T23:53:53.995 INFO:teuthology.orchestra.run.smithi102.stdout:(135/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-03-08T23:53:54.070 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:53:54.177 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T23:53:54.187 INFO:teuthology.orchestra.run.smithi102.stdout:(136/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 469 kB/s | 90 kB 00:00 2024-03-08T23:53:54.272 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T23:53:54.296 INFO:teuthology.orchestra.run.smithi102.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 593 kB/s | 272 kB 00:00 2024-03-08T23:53:54.311 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T23:53:54.321 INFO:teuthology.orchestra.run.smithi102.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 211 kB/s | 28 kB 00:00 2024-03-08T23:53:54.338 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T23:53:54.373 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T23:53:54.438 INFO:teuthology.orchestra.run.smithi102.stdout:(139/174): libarrow-9.0.0-10.el9.x86_64.rpm 808 kB/s | 4.4 MB 00:05 2024-03-08T23:53:54.445 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T23:53:54.463 INFO:teuthology.orchestra.run.smithi102.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 184 kB/s | 31 kB 00:00 2024-03-08T23:53:54.500 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T23:53:54.555 INFO:teuthology.orchestra.run.smithi102.stdout:(141/174): python3-rsa-4.9-2.el9.noarch.rpm 506 kB/s | 59 kB 00:00 2024-03-08T23:53:54.556 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T23:53:54.580 INFO:teuthology.orchestra.run.smithi102.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 306 kB/s | 36 kB 00:00 2024-03-08T23:53:54.621 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T23:53:54.622 INFO:teuthology.orchestra.run.smithi102.stdout:(143/174): python3-routes-2.5.1-5.el9.noarch.rp 626 kB/s | 188 kB 00:00 2024-03-08T23:53:54.656 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T23:53:54.656 INFO:teuthology.orchestra.run.smithi102.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 507 kB/s | 51 kB 00:00 2024-03-08T23:53:54.699 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T23:53:54.732 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T23:53:54.760 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T23:53:54.814 INFO:teuthology.orchestra.run.smithi102.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 467 kB/s | 90 kB 00:00 2024-03-08T23:53:54.828 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T23:53:54.865 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T23:53:54.908 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T23:53:54.923 INFO:teuthology.orchestra.run.smithi102.stdout:(146/174): python3-zc-lockfile-2.0-10.el9.noarc 184 kB/s | 20 kB 00:00 2024-03-08T23:53:54.936 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T23:53:54.948 INFO:teuthology.orchestra.run.smithi102.stdout:(147/174): python3-webob-1.8.7-6.el9.noarch.rpm 624 kB/s | 230 kB 00:00 2024-03-08T23:53:55.006 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T23:53:55.059 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T23:53:55.133 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T23:53:55.190 INFO:teuthology.orchestra.run.smithi102.stdout:(148/174): re2-20211101-3.el9.x86_64.rpm 718 kB/s | 192 kB 00:00 2024-03-08T23:53:55.232 INFO:teuthology.orchestra.run.smithi102.stdout:(149/174): python3-werkzeug-2.0.3-3.el9.1.noarc 741 kB/s | 427 kB 00:00 2024-03-08T23:53:55.264 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T23:53:55.320 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T23:53:55.533 INFO:teuthology.orchestra.run.smithi102.stdout:(150/174): librbd1-19.0.0-1965.g7d63482a.el9.x8 10 MB/s | 3.1 MB 00:00 2024-03-08T23:53:55.583 INFO:teuthology.orchestra.run.smithi102.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 6.3 MB/s | 321 kB 00:00 2024-03-08T23:53:55.832 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T23:53:55.917 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T23:53:55.967 INFO:teuthology.orchestra.run.smithi102.stdout:(152/174): NetworkManager-1.46.0-1.el9.x86_64.r 6.0 MB/s | 2.3 MB 00:00 2024-03-08T23:53:55.993 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T23:53:56.037 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:53:56.045 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:53:56.077 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T23:53:56.123 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T23:53:56.163 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T23:53:56.167 INFO:teuthology.orchestra.run.smithi102.stdout:(153/174): librados2-19.0.0-1965.g7d63482a.el9. 3.5 MB/s | 3.4 MB 00:00 2024-03-08T23:53:56.194 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T23:53:56.217 INFO:teuthology.orchestra.run.smithi102.stdout:(154/174): NetworkManager-libnm-1.46.0-1.el9.x8 7.3 MB/s | 1.8 MB 00:00 2024-03-08T23:53:56.243 INFO:teuthology.orchestra.run.smithi102.stdout:(155/174): NetworkManager-team-1.46.0-1.el9.x86 564 kB/s | 42 kB 00:00 2024-03-08T23:53:56.276 INFO:teuthology.orchestra.run.smithi102.stdout:(156/174): NetworkManager-tui-1.46.0-1.el9.x86_ 4.1 MB/s | 248 kB 00:00 2024-03-08T23:53:56.301 INFO:teuthology.orchestra.run.smithi102.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-08T23:53:56.327 INFO:teuthology.orchestra.run.smithi102.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 725 kB/s | 36 kB 00:00 2024-03-08T23:53:56.386 INFO:teuthology.orchestra.run.smithi102.stdout:(159/174): elfutils-libs-0.190-2.el9.x86_64.rpm 4.3 MB/s | 256 kB 00:00 2024-03-08T23:53:56.411 INFO:teuthology.orchestra.run.smithi102.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.7 MB/s | 192 kB 00:00 2024-03-08T23:53:56.428 INFO:teuthology.orchestra.run.smithi102.stdout:(161/174): libacl-2.3.1-4.el9.x86_64.rpm 550 kB/s | 23 kB 00:00 2024-03-08T23:53:56.457 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T23:53:56.461 INFO:teuthology.orchestra.run.smithi102.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 2.1 MB/s | 108 kB 00:00 2024-03-08T23:53:56.486 INFO:teuthology.orchestra.run.smithi102.stdout:(163/174): libfdisk-2.37.4-18.el9.x86_64.rpm 2.6 MB/s | 155 kB 00:00 2024-03-08T23:53:56.520 INFO:teuthology.orchestra.run.smithi102.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 2.3 MB/s | 136 kB 00:00 2024-03-08T23:53:56.537 INFO:teuthology.orchestra.run.smithi102.stdout:(165/174): libsmartcols-2.37.4-18.el9.x86_64.rp 1.2 MB/s | 64 kB 00:00 2024-03-08T23:53:56.562 INFO:teuthology.orchestra.run.smithi102.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 684 kB/s | 29 kB 00:00 2024-03-08T23:53:56.611 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T23:53:56.662 INFO:teuthology.orchestra.run.smithi102.stdout:(167/174): openssl-3.0.7-27.el9.x86_64.rpm 9.7 MB/s | 1.2 MB 00:00 2024-03-08T23:53:56.743 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T23:53:56.855 INFO:teuthology.orchestra.run.smithi102.stdout:(168/174): util-linux-2.37.4-18.el9.x86_64.rpm 12 MB/s | 2.3 MB 00:00 2024-03-08T23:53:56.922 INFO:teuthology.orchestra.run.smithi102.stdout:(169/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 6.0 MB/s | 2.2 MB 00:00 2024-03-08T23:53:56.944 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T23:53:56.947 INFO:teuthology.orchestra.run.smithi102.stdout:(170/174): util-linux-core-2.37.4-18.el9.x86_64 4.9 MB/s | 465 kB 00:00 2024-03-08T23:53:56.964 INFO:teuthology.orchestra.run.smithi102.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 743 kB/s | 31 kB 00:00 2024-03-08T23:53:57.014 INFO:teuthology.orchestra.run.smithi102.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 74 kB 00:00 2024-03-08T23:53:57.048 INFO:teuthology.orchestra.run.smithi102.stdout:(173/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 11 MB/s | 1.1 MB 00:00 2024-03-08T23:53:57.078 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T23:53:57.089 INFO:teuthology.orchestra.run.smithi102.stdout:(174/174): thrift-0.15.0-2.el9.x86_64.rpm 756 kB/s | 1.6 MB 00:02 2024-03-08T23:53:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:-------------------------------------------------------------------------------- 2024-03-08T23:53:57.097 INFO:teuthology.orchestra.run.smithi102.stdout:Total 12 MB/s | 223 MB 00:18 2024-03-08T23:53:57.127 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T23:53:57.183 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T23:53:57.255 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T23:53:57.286 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T23:53:57.314 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T23:53:57.362 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T23:53:57.398 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T23:53:57.441 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T23:53:57.483 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T23:53:57.571 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction check 2024-03-08T23:53:57.616 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T23:53:57.681 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T23:53:57.719 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T23:53:57.761 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T23:53:57.838 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction check succeeded. 2024-03-08T23:53:57.838 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction test 2024-03-08T23:53:57.854 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T23:53:57.896 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T23:53:57.936 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T23:53:57.978 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T23:53:58.002 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T23:53:58.081 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T23:53:58.204 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T23:53:58.312 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T23:53:58.361 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T23:53:58.410 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T23:53:58.465 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T23:53:58.648 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T23:53:58.699 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T23:53:58.833 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T23:53:58.872 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T23:53:58.903 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T23:53:58.934 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T23:53:58.962 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T23:53:58.997 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T23:53:59.035 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T23:53:59.068 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T23:53:59.097 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T23:53:59.173 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T23:53:59.208 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T23:53:59.241 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T23:53:59.283 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T23:53:59.317 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T23:53:59.406 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T23:53:59.488 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T23:53:59.531 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T23:53:59.539 INFO:teuthology.orchestra.run.smithi102.stdout:Transaction test succeeded. 2024-03-08T23:53:59.540 INFO:teuthology.orchestra.run.smithi102.stdout:Running transaction 2024-03-08T23:53:59.563 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T23:53:59.610 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T23:53:59.707 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T23:53:59.756 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T23:54:00.475 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T23:54:00.528 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T23:54:00.562 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T23:54:00.610 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T23:54:00.697 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T23:54:00.746 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T23:54:00.922 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T23:54:00.954 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T23:54:01.031 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T23:54:01.269 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:54:01.269 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /sys 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /proc 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /mnt 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /var/tmp 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /home 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /root 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout:skipping the directory /tmp 2024-03-08T23:54:01.270 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:01.432 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:01.448 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:01.448 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:01.448 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T23:54:01.448 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:01.667 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T23:54:01.775 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T23:54:02.938 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:02.962 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:02.963 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:02.963 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T23:54:02.963 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:02.963 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:02.963 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:02.990 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T23:54:03.063 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T23:54:03.094 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T23:54:03.125 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T23:54:03.152 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T23:54:03.170 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:03.193 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:03.468 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T23:54:03.629 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:03.651 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:03.651 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T23:54:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:03.652 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:03.652 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:03.707 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:54:03.756 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T23:54:03.813 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:54:03.854 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T23:54:03.890 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T23:54:04.090 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:04.121 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:04.153 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T23:54:04.213 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T23:54:04.236 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:04.346 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T23:54:04.377 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:54:04.432 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:54:04.454 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T23:54:04.470 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:04.504 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:05.469 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:05.495 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:05.596 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:05.759 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T23:54:05.789 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:05.811 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:05.811 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:05.812 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T23:54:05.812 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:05.812 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:05.812 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:05.857 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:05.867 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:05.913 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T23:54:06.446 INFO:teuthology.orchestra.run.smithi102.stdout: Preparing : 1/1 2024-03-08T23:54:06.562 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-08T23:54:06.605 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-08T23:54:06.607 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:06.636 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:54:06.645 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-08T23:54:06.650 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:06.693 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-08T23:54:06.755 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-08T23:54:06.774 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T23:54:06.797 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-08T23:54:06.800 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:06.823 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:06.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-08T23:54:06.868 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:06.873 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-08T23:54:06.890 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:06.890 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:06.890 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T23:54:06.890 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:06.935 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:06.957 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:06.957 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:06.958 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T23:54:06.958 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:06.958 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:06.958 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:07.186 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:07.189 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-08T23:54:07.220 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:54:07.240 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:54:07.269 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:07.291 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:07.291 INFO:teuthology.orchestra.run.smithi179.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:07.292 INFO:teuthology.orchestra.run.smithi179.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T23:54:07.292 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:07.292 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:07.292 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:07.313 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:54:07.313 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T23:54:07.313 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T23:54:07.313 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:07.346 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:54:07.362 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:54:07.392 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:54:07.483 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:54:07.513 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:54:07.547 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-08T23:54:07.710 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-08T23:54:07.760 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-08T23:54:07.797 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:54:07.884 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:54:07.915 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:54:07.962 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 17/198 2024-03-08T23:54:08.002 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-08T23:54:08.025 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:54:08.040 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-08T23:54:08.241 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:54:08.281 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-08T23:54:08.321 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-08T23:54:08.434 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-08T23:54:08.548 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:54:08.608 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 23/198 2024-03-08T23:54:08.678 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:54:08.734 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:54:09.216 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.46.0-1.el9.x86_64 24/198 2024-03-08T23:54:09.327 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-08T23:54:09.423 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-08T23:54:09.457 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-08T23:54:09.484 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-08T23:54:09.528 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-08T23:54:09.608 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-08T23:54:09.690 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-08T23:54:09.736 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-08T23:54:09.800 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-08T23:54:09.834 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-08T23:54:09.878 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-08T23:54:09.911 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-08T23:54:09.939 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-08T23:54:10.009 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-08T23:54:10.044 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 39/198 2024-03-08T23:54:10.087 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 40/198 2024-03-08T23:54:10.124 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 41/198 2024-03-08T23:54:10.193 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-08T23:54:10.239 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-08T23:54:10.311 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-08T23:54:10.445 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-08T23:54:10.507 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-08T23:54:11.038 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-08T23:54:11.120 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-08T23:54:11.191 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-08T23:54:11.240 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:54:11.248 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-08T23:54:11.280 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-08T23:54:11.320 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-08T23:54:11.367 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 53/198 2024-03-08T23:54:11.405 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-08T23:54:11.672 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-08T23:54:11.707 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:11.748 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T23:54:11.779 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T23:54:11.806 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T23:54:11.847 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-08T23:54:11.989 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T23:54:12.016 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-08T23:54:12.107 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T23:54:12.145 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T23:54:12.179 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-08T23:54:12.184 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T23:54:12.246 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T23:54:12.289 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T23:54:12.322 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T23:54:12.322 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-08T23:54:12.362 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T23:54:12.364 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-08T23:54:12.392 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T23:54:12.419 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-08T23:54:12.437 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T23:54:12.492 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-08T23:54:12.498 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T23:54:12.523 INFO:teuthology.orchestra.run.smithi179.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T23:54:12.524 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-08T23:54:12.569 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-08T23:54:12.615 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-08T23:54:12.651 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-08T23:54:12.694 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-08T23:54:12.737 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-08T23:54:12.869 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-08T23:54:12.918 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-08T23:54:12.947 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-08T23:54:12.989 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-08T23:54:13.082 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-08T23:54:13.131 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-google-auth-1:2.28.0-1.el9.noarch 74/198 2024-03-08T23:54:13.145 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:13.170 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-08T23:54:13.180 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:13.196 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-08T23:54:13.199 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:13.221 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-08T23:54:13.309 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-08T23:54:13.438 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-08T23:54:13.548 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-08T23:54:13.588 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-08T23:54:13.636 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-08T23:54:13.692 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-08T23:54:13.885 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-08T23:54:13.934 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-08T23:54:14.016 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T23:54:14.062 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-08T23:54:14.099 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-asyncssh-2.13.2-4.el9.noarch 87/198 2024-03-08T23:54:14.100 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T23:54:14.130 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-08T23:54:14.141 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T23:54:14.162 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-08T23:54:14.190 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-08T23:54:14.222 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T23:54:14.232 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-08T23:54:14.254 INFO:teuthology.orchestra.run.smithi179.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T23:54:14.278 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T23:54:14.280 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-08T23:54:14.304 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T23:54:14.313 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-08T23:54:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T23:54:14.330 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:14.341 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-08T23:54:14.417 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-08T23:54:14.418 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T23:54:14.448 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:14.460 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-08T23:54:14.494 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-08T23:54:14.516 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:14.535 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-08T23:54:14.542 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T23:54:14.569 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-08T23:54:14.575 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T23:54:14.601 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T23:54:14.626 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T23:54:14.659 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T23:54:14.659 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-08T23:54:14.698 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T23:54:14.731 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T23:54:14.750 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-08T23:54:14.756 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T23:54:14.781 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T23:54:14.800 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-08T23:54:14.806 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:14.831 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T23:54:14.832 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-08T23:54:14.837 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:14.838 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:14.878 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-08T23:54:14.977 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-08T23:54:15.018 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:15.018 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:15.025 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-08T23:54:15.101 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:15.126 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:15.151 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T23:54:15.191 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T23:54:15.218 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T23:54:15.282 INFO:teuthology.orchestra.run.smithi179.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:15.752 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-08T23:54:15.797 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-08T23:54:15.831 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-08T23:54:15.880 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-08T23:54:15.968 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-08T23:54:15.999 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-08T23:54:16.174 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-08T23:54:16.213 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-08T23:54:16.299 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-08T23:54:16.944 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-08T23:54:17.070 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-08T23:54:18.299 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-08T23:54:18.389 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-08T23:54:18.445 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-08T23:54:18.485 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-08T23:54:18.512 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-08T23:54:18.822 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-08T23:54:18.859 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:54:18.940 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 124/198 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /sys 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /proc 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /mnt 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /var/tmp 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /home 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /root 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout:skipping the directory /tmp 2024-03-08T23:54:18.956 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:18.972 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 125/198 2024-03-08T23:54:19.008 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-08T23:54:19.119 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:19.133 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:19.133 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:19.133 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T23:54:19.133 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:19.458 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-08T23:54:19.487 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:54:19.558 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 128/198 2024-03-08T23:54:19.580 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 129/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T23:54:20.240 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T23:54:20.241 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T23:54:20.242 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T23:54:20.243 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T23:54:20.244 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T23:54:20.245 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T23:54:20.246 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T23:54:20.247 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T23:54:20.248 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T23:54:20.249 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T23:54:20.250 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T23:54:20.251 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T23:54:20.252 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:20.253 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T23:54:20.254 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T23:54:20.647 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:20.672 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:20.872 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:20.894 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:20.894 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:20.895 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T23:54:20.895 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:20.895 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:20.895 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:21.348 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:21.370 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:21.370 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:21.370 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T23:54:21.370 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:21.371 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:21.371 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:21.458 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T23:54:21.714 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:21.744 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:21.833 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:21.865 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:21.889 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T23:54:21.940 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T23:54:21.963 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:22.195 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:22.222 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:22.274 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 130/198 2024-03-08T23:54:22.322 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:54:22.395 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 131/198 2024-03-08T23:54:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-08T23:54:22.395 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-08T23:54:22.395 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:22.423 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:54:23.191 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:23.223 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:23.323 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:23.484 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T23:54:23.495 INFO:teuthology.orchestra.run.smithi179.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout:Upgraded: 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T23:54:23.496 INFO:teuthology.orchestra.run.smithi179.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T23:54:23.497 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout:Installed: 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.498 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.499 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: jq-1.6-16.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.500 INFO:teuthology.orchestra.run.smithi179.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T23:54:23.501 INFO:teuthology.orchestra.run.smithi179.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T23:54:23.502 INFO:teuthology.orchestra.run.smithi179.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T23:54:23.503 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T23:54:23.504 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T23:54:23.505 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T23:54:23.506 INFO:teuthology.orchestra.run.smithi179.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T23:54:23.507 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T23:54:23.508 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T23:54:23.509 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:23.509 INFO:teuthology.orchestra.run.smithi179.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T23:54:23.509 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:54:23.509 INFO:teuthology.orchestra.run.smithi179.stdout:Complete! 2024-03-08T23:54:23.530 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:23.530 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:23.531 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T23:54:23.531 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:23.531 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:23.531 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:23.575 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:23.586 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:23.633 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T23:54:24.001 DEBUG:teuthology.parallel:result is None 2024-03-08T23:54:24.498 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T23:54:24.536 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:24.559 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:24.603 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:24.624 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:24.624 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:24.624 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T23:54:24.625 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:24.670 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:24.692 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:24.692 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:24.692 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T23:54:24.693 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:24.693 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:24.693 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:24.996 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:25.019 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:29.459 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:29.517 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T23:54:29.547 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T23:54:29.574 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T23:54:29.763 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T23:54:29.873 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T23:54:29.914 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T23:54:29.965 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T23:54:30.039 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T23:54:30.074 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T23:54:30.107 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T23:54:30.156 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T23:54:30.186 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T23:54:30.237 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T23:54:30.291 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T23:54:30.317 INFO:teuthology.orchestra.run.smithi073.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T23:54:30.897 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:30.948 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:30.967 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:31.774 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T23:54:31.844 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T23:54:31.893 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T23:54:31.986 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T23:54:32.022 INFO:teuthology.orchestra.run.smithi073.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T23:54:32.047 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T23:54:32.073 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T23:54:32.102 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T23:54:32.102 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:32.187 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:32.225 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T23:54:32.225 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:32.310 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:32.367 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T23:54:32.402 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T23:54:32.428 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T23:54:32.453 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T23:54:32.486 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T23:54:32.536 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T23:54:32.574 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T23:54:32.600 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T23:54:32.625 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T23:54:32.650 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:32.675 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T23:54:32.681 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:32.681 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:32.862 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:32.862 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:32.953 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:32.978 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:33.003 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T23:54:33.036 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T23:54:33.071 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T23:54:33.137 INFO:teuthology.orchestra.run.smithi073.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:34.146 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 132/198 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /sys 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /proc 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /mnt 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /var/tmp 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /home 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /root 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout:skipping the directory /tmp 2024-03-08T23:54:34.147 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:34.312 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:34.327 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 133/198 2024-03-08T23:54:34.327 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:34.327 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-08T23:54:34.327 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:35.790 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 134/198 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-08T23:54:35.816 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:36.024 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:36.046 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 135/198 2024-03-08T23:54:36.047 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:36.047 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-08T23:54:36.047 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:36.047 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-08T23:54:36.047 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:36.483 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 136/198 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-08T23:54:36.506 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:36.593 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-08T23:54:36.952 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:36.984 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 138/198 2024-03-08T23:54:37.017 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 139/198 2024-03-08T23:54:37.075 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 140/198 2024-03-08T23:54:37.098 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:37.330 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 141/198 2024-03-08T23:54:37.366 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:38.077 INFO:teuthology.orchestra.run.smithi073.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:38.077 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:54:38.078 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T23:54:38.079 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T23:54:38.080 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T23:54:38.080 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T23:54:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T23:54:38.095 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T23:54:38.096 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T23:54:38.097 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T23:54:38.098 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T23:54:38.099 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T23:54:38.100 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T23:54:38.101 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T23:54:38.102 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T23:54:38.103 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:38.104 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T23:54:38.105 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T23:54:38.106 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:38.107 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T23:54:38.108 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T23:54:38.322 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 142/198 2024-03-08T23:54:38.350 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:38.449 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 143/198 2024-03-08T23:54:38.613 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 144/198 2024-03-08T23:54:38.652 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 145/198 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-08T23:54:38.675 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:38.719 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:38.730 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 146/198 2024-03-08T23:54:38.775 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 147/198 2024-03-08T23:54:39.634 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-08T23:54:39.663 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 149/198 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-08T23:54:39.686 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:39.730 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-1965.g7d634 150/198 2024-03-08T23:54:39.752 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:39.752 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-08T23:54:39.752 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:39.798 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-08T23:54:39.820 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:40.123 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 152/198 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-08T23:54:40.146 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:40.962 INFO:teuthology.orchestra.run.smithi073.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T23:54:40.962 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:40.962 INFO:teuthology.orchestra.run.smithi073.stdout:Upgraded: 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.963 INFO:teuthology.orchestra.run.smithi073.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.964 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout:Installed: 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.965 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.966 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T23:54:40.967 INFO:teuthology.orchestra.run.smithi073.stdout: jq-1.6-16.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.968 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:40.969 INFO:teuthology.orchestra.run.smithi073.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T23:54:40.970 INFO:teuthology.orchestra.run.smithi073.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T23:54:40.971 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T23:54:40.972 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T23:54:40.973 INFO:teuthology.orchestra.run.smithi073.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.974 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T23:54:40.975 INFO:teuthology.orchestra.run.smithi073.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:40.976 INFO:teuthology.orchestra.run.smithi073.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T23:54:40.977 INFO:teuthology.orchestra.run.smithi073.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T23:54:40.977 INFO:teuthology.orchestra.run.smithi073.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T23:54:40.977 INFO:teuthology.orchestra.run.smithi073.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T23:54:40.977 INFO:teuthology.orchestra.run.smithi073.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T23:54:40.977 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T23:54:40.978 INFO:teuthology.orchestra.run.smithi073.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T23:54:40.978 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:54:40.978 INFO:teuthology.orchestra.run.smithi073.stdout:Complete! 2024-03-08T23:54:41.468 DEBUG:teuthology.parallel:result is None 2024-03-08T23:54:44.536 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:44.579 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 154/198 2024-03-08T23:54:44.609 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 155/198 2024-03-08T23:54:44.637 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-08T23:54:44.814 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 157/198 2024-03-08T23:54:44.946 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-08T23:54:44.987 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 159/198 2024-03-08T23:54:45.024 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-08T23:54:45.077 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-08T23:54:45.128 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-08T23:54:45.169 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 163/198 2024-03-08T23:54:45.202 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-08T23:54:45.231 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-team-1:1.46.0-1.el9.x86_64 165/198 2024-03-08T23:54:45.301 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : NetworkManager-tui-1:1.46.0-1.el9.x86_64 166/198 2024-03-08T23:54:45.361 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-08T23:54:45.396 INFO:teuthology.orchestra.run.smithi102.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-08T23:54:45.981 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:46.020 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:46.039 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-08T23:54:46.858 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 170/198 2024-03-08T23:54:46.948 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-08T23:54:46.989 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-08T23:54:47.080 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-08T23:54:47.110 INFO:teuthology.orchestra.run.smithi102.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-08T23:54:47.135 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-08T23:54:47.160 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-08T23:54:47.187 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-08T23:54:47.187 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:47.275 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-08T23:54:47.313 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-08T23:54:47.313 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:47.381 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-08T23:54:47.407 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-08T23:54:47.440 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-08T23:54:47.466 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-08T23:54:47.491 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-08T23:54:47.524 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-08T23:54:47.563 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-08T23:54:47.596 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-08T23:54:47.621 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-08T23:54:47.646 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-08T23:54:47.671 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:47.697 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-08T23:54:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:47.703 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:47.891 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-08T23:54:47.892 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:47.974 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-08T23:54:47.999 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:48.025 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-08T23:54:48.058 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-08T23:54:48.084 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-08T23:54:48.149 INFO:teuthology.orchestra.run.smithi102.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:53.025 INFO:teuthology.orchestra.run.smithi102.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-08T23:54:53.025 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 1/198 2024-03-08T23:54:53.025 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2/198 2024-03-08T23:54:53.025 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 3/198 2024-03-08T23:54:53.025 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 4/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-1965.g7d634 5/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 6/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 7/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 8/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 9/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 10/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 11/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 12/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 13/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 14/198 2024-03-08T23:54:53.026 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_ 15/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 16/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 17/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_6 18/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86 19/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 20/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el 21/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ceph-common-2:19.0.0-1965.g7d63482a.el9. 22/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_6 23/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 24/198 2024-03-08T23:54:53.027 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 25/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 26/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 27/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 28/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 29/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a. 30/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noa 31/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.n 32/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d6 33/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el 34/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 35/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.e 36/198 2024-03-08T23:54:53.028 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 37/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 38/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 39/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-08T23:54:53.029 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-08T23:54:53.030 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-08T23:54:53.031 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-08T23:54:53.032 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-08T23:54:53.033 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-08T23:54:53.034 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-08T23:54:53.035 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-asyncssh-2.13.2-4.el9.noarch 116/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-08T23:54:53.036 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-google-auth-1:2.28.0-1.el9.noarch 124/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-08T23:54:53.037 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-08T23:54:53.038 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 151/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 153/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-08T23:54:53.039 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.46.0-1.el9.x86_64 157/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.46.0-1.el9.x86_64 159/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.46.0-1.el9.x86_64 161/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.46.0-1.el9.x86_64 163/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-08T23:54:53.040 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-08T23:54:53.041 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-08T23:54:53.042 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-08T23:54:53.043 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-08T23:54:53.043 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-08T23:54:53.043 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-08T23:54:53.043 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-08T23:54:53.043 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-08T23:54:55.889 INFO:teuthology.orchestra.run.smithi102.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-08T23:54:55.889 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:55.889 INFO:teuthology.orchestra.run.smithi102.stdout:Upgraded: 2024-03-08T23:54:55.889 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-libnm-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-team-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: NetworkManager-tui-1:1.46.0-1.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: acl-2.3.1-4.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.890 INFO:teuthology.orchestra.run.smithi102.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: librados2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: librbd1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.891 INFO:teuthology.orchestra.run.smithi102.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout:Installed: 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: automake-1.16.2-8.el9.noarch 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: bison-3.7.4-5.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-base-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.892 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-grafana-dashboards-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-immutable-object-cache-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mds-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-dashboard-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-diskprediction-local-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-modules-core-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mgr-rook-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-mon-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-osd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-prometheus-alerts-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.893 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-radosgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-selinux-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-test-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: ceph-volume-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: cephadm-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-shell-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: cephfs-top-2:19.0.0-1965.g7d63482a.el9.noarch 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: flex-2.6.4-9.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-08T23:54:55.894 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: jq-1.6-16.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libcephfs2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.895 INFO:teuthology.orchestra.run.smithi102.stdout: libcephsqlite-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: librados-devel-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libradosstriper1-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: librgw2-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-08T23:54:55.896 INFO:teuthology.orchestra.run.smithi102.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: lua-5.4.4-4.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-13.11-1.el9.x86_64 2024-03-08T23:54:55.897 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-asyncssh-2.13.2-4.el9.noarch 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-argparse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.898 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ceph-common-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cephfs-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-google-auth-1:2.28.0-1.el9.noarch 2024-03-08T23:54:55.899 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-08T23:54:55.900 INFO:teuthology.orchestra.run.smithi102.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-ply-3.11-14.el9.noarch 2024-03-08T23:54:55.901 INFO:teuthology.orchestra.run.smithi102.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rados-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.902 INFO:teuthology.orchestra.run.smithi102.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rgw-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-08T23:54:55.903 INFO:teuthology.orchestra.run.smithi102.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-fuse-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-mirror-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.904 INFO:teuthology.orchestra.run.smithi102.stdout: rbd-nbd-2:19.0.0-1965.g7d63482a.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: re2-1:20211101-3.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-08T23:54:55.905 INFO:teuthology.orchestra.run.smithi102.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-08T23:54:55.906 INFO:teuthology.orchestra.run.smithi102.stdout: 2024-03-08T23:54:55.906 INFO:teuthology.orchestra.run.smithi102.stdout:Complete! 2024-03-08T23:54:56.398 DEBUG:teuthology.parallel:result is None 2024-03-08T23:54:56.399 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:56.630 DEBUG:teuthology.orchestra.run.smithi073:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T23:54:56.657 INFO:teuthology.orchestra.run.smithi073.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:56.657 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:56.657 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T23:54:56.660 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:56.798 DEBUG:teuthology.orchestra.run.smithi102:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T23:54:56.821 INFO:teuthology.orchestra.run.smithi102.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:56.821 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:56.821 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T23:54:56.823 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:57.039 DEBUG:teuthology.orchestra.run.smithi179:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-08T23:54:57.065 INFO:teuthology.orchestra.run.smithi179.stdout:19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:57.065 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-1965.g7d63482a.el9 2024-03-08T23:54:57.065 INFO:teuthology.task.install:The correct ceph version 19.0.0-1965.g7d63482a is installed. 2024-03-08T23:54:57.067 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-08T23:54:57.067 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:54:57.067 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T23:54:57.102 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:54:57.102 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T23:54:57.134 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:54:57.134 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-08T23:54:57.168 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-08T23:54:57.169 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:54:57.169 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T23:54:57.200 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T23:54:57.270 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:54:57.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T23:54:57.299 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T23:54:57.366 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:54:57.367 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2024-03-08T23:54:57.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-08T23:54:57.470 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-08T23:54:57.470 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:54:57.470 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T23:54:57.497 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T23:54:57.565 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:54:57.565 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T23:54:57.592 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T23:54:57.660 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:54:57.660 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-08T23:54:57.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-08T23:54:57.753 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-08T23:54:57.753 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:54:57.753 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T23:54:57.780 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T23:54:57.847 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:54:57.847 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T23:54:57.873 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T23:54:57.941 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:54:57.942 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/stdin-killer 2024-03-08T23:54:57.968 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-08T23:54:58.035 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-08T23:54:58.147 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug cephsqlite': 20, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-08T23:54:58.147 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:58.148 INFO:tasks.cephadm:Cluster fsid is 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:54:58.148 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-08T23:54:58.148 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.73', 'mon.b': '172.21.15.102', 'mon.c': '172.21.15.179'} 2024-03-08T23:54:58.148 INFO:tasks.cephadm:First mon is mon.a on smithi073 2024-03-08T23:54:58.148 INFO:tasks.cephadm:First mgr is x 2024-03-08T23:54:58.148 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-08T23:54:58.148 DEBUG:teuthology.orchestra.run.smithi073:> sudo hostname $(hostname -s) 2024-03-08T23:54:58.175 DEBUG:teuthology.orchestra.run.smithi102:> sudo hostname $(hostname -s) 2024-03-08T23:54:58.203 DEBUG:teuthology.orchestra.run.smithi179:> sudo hostname $(hostname -s) 2024-03-08T23:54:58.230 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-08T23:54:58.230 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:58.445 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '7d63482aa89628d733e298984a762389b625c761', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77443/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-1965-g7d63482a', 'node_name': '172.21.2.1+braggi01', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-1965.g7d63482a'}, 'url': 'https://1.chacra.ceph.com/r/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-08 17:09:11.137784', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'rishabh-2024mar8', 'chacra_url': 'https://1.chacra.ceph.com/repos/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-08T23:54:58.584 INFO:tasks.util.chacra:got chacra host 1.chacra.ceph.com, ref rishabh-2024mar8, sha1 7d63482aa89628d733e298984a762389b625c761 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:54:58.585 INFO:tasks.cephadm:Discovered cachra url: https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm 2024-03-08T23:54:58.586 DEBUG:teuthology.orchestra.run.smithi073:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T23:54:59.190 INFO:teuthology.orchestra.run.smithi073.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 23:54 /home/ubuntu/cephtest/cephadm 2024-03-08T23:54:59.190 DEBUG:teuthology.orchestra.run.smithi102:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T23:54:59.654 INFO:teuthology.orchestra.run.smithi102.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 23:54 /home/ubuntu/cephtest/cephadm 2024-03-08T23:54:59.654 DEBUG:teuthology.orchestra.run.smithi179:> curl --silent -L https://1.chacra.ceph.com/binaries/ceph/rishabh-2024mar8/7d63482aa89628d733e298984a762389b625c761/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-08T23:55:00.215 INFO:teuthology.orchestra.run.smithi179.stdout:-rw-r--r--. 1 ubuntu ubuntu 767402 Mar 8 23:55 /home/ubuntu/cephtest/cephadm 2024-03-08T23:55:00.215 DEBUG:teuthology.orchestra.run.smithi073:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T23:55:00.234 DEBUG:teuthology.orchestra.run.smithi102:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T23:55:00.252 DEBUG:teuthology.orchestra.run.smithi179:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-08T23:55:00.287 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 on all hosts... 2024-03-08T23:55:00.287 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T23:55:00.290 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T23:55:00.295 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 pull 2024-03-08T23:55:01.076 INFO:teuthology.orchestra.run.smithi179.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T23:55:01.175 INFO:teuthology.orchestra.run.smithi073.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T23:55:01.214 INFO:teuthology.orchestra.run.smithi102.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T23:55:34.106 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "repo_digests": [ 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout: ] 2024-03-08T23:55:34.107 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-08T23:55:48.130 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-08T23:55:48.130 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T23:55:48.131 INFO:teuthology.orchestra.run.smithi102.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T23:55:48.131 INFO:teuthology.orchestra.run.smithi102.stdout: "repo_digests": [ 2024-03-08T23:55:48.131 INFO:teuthology.orchestra.run.smithi102.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T23:55:48.131 INFO:teuthology.orchestra.run.smithi102.stdout: ] 2024-03-08T23:55:48.131 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-08T23:55:48.436 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout: "image_id": "a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249", 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout: "repo_digests": [ 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd" 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-08T23:55:48.437 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-08T23:55:48.460 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph 2024-03-08T23:55:48.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo mkdir -p /etc/ceph 2024-03-08T23:55:48.526 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph 2024-03-08T23:55:48.561 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /etc/ceph 2024-03-08T23:55:48.588 DEBUG:teuthology.orchestra.run.smithi102:> sudo chmod 777 /etc/ceph 2024-03-08T23:55:48.614 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /etc/ceph 2024-03-08T23:55:48.644 INFO:tasks.cephadm:Writing seed config... 2024-03-08T23:55:48.645 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-08T23:55:48.645 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-08T23:55:48.645 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-08T23:55:48.646 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mgr] debug cephsqlite = 20 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-08T23:55:48.647 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-08T23:55:48.648 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-08T23:55:48.649 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:55:48.649 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-08T23:55:48.672 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 44f886a8-dda7-11ee-95c6-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug cephsqlite = 20 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-08T23:55:48.673 DEBUG:teuthology.orchestra.run.smithi073:mon.a> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service 2024-03-08T23:55:48.717 DEBUG:teuthology.orchestra.run.smithi073:mgr.x> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service 2024-03-08T23:55:48.766 INFO:tasks.cephadm:Bootstrapping... 2024-03-08T23:55:48.766 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 -v bootstrap --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.73 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:55:48.960 INFO:teuthology.orchestra.run.smithi073.stdout:-------------------------------------------------------------------------------- 2024-03-08T23:55:48.960 INFO:teuthology.orchestra.run.smithi073.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761', '-v', 'bootstrap', '--fsid', '44f886a8-dda7-11ee-95c6-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.73', '--skip-admin-label'] 2024-03-08T23:55:48.985 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T23:55:48.986 INFO:teuthology.orchestra.run.smithi073.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-08T23:55:48.986 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying podman|docker is present... 2024-03-08T23:55:49.009 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T23:55:49.009 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying lvm2 is present... 2024-03-08T23:55:49.009 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying time synchronization is in place... 2024-03-08T23:55:49.016 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-08T23:55:49.016 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-08T23:55:49.022 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-08T23:55:49.022 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:55:49.029 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout enabled 2024-03-08T23:55:49.034 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout active 2024-03-08T23:55:49.034 INFO:teuthology.orchestra.run.smithi073.stdout:Unit chronyd.service is enabled and running 2024-03-08T23:55:49.035 INFO:teuthology.orchestra.run.smithi073.stdout:Repeating the final host check... 2024-03-08T23:55:49.058 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-08T23:55:49.058 INFO:teuthology.orchestra.run.smithi073.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-08T23:55:49.058 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl is present 2024-03-08T23:55:49.058 INFO:teuthology.orchestra.run.smithi073.stdout:lvcreate is present 2024-03-08T23:55:49.064 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-08T23:55:49.065 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-08T23:55:49.070 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-08T23:55:49.070 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:55:49.077 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout enabled 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout active 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Unit chronyd.service is enabled and running 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Host looks OK 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Cluster fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Acquiring lock 140387904359872 on /run/cephadm/44f886a8-dda7-11ee-95c6-87774f69a715.lock 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Lock 140387904359872 acquired on /run/cephadm/44f886a8-dda7-11ee-95c6-87774f69a715.lock 2024-03-08T23:55:49.083 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying IP 172.21.15.73 port 3300 ... 2024-03-08T23:55:49.084 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying IP 172.21.15.73 port 6789 ... 2024-03-08T23:55:49.084 INFO:teuthology.orchestra.run.smithi073.stdout:Base mon IP(s) is [172.21.15.73:3300, 172.21.15.73:6789], mon addrv is [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-03-08T23:55:49.087 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.73 metric 100 2024-03-08T23:55:49.087 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.73 metric 100 2024-03-08T23:55:49.090 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-08T23:55:49.090 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-08T23:55:49.090 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1756sec hoplimit 64 pref medium 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-03-08T23:55:49.092 INFO:teuthology.orchestra.run.smithi073.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-08T23:55:49.093 INFO:teuthology.orchestra.run.smithi073.stdout:Mon IP `172.21.15.73` is in CIDR network `172.21.0.0/20` 2024-03-08T23:55:49.093 INFO:teuthology.orchestra.run.smithi073.stdout:Mon IP `172.21.15.73` is in CIDR network `172.21.0.0/20` 2024-03-08T23:55:49.093 INFO:teuthology.orchestra.run.smithi073.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-08T23:55:49.093 INFO:teuthology.orchestra.run.smithi073.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-08T23:55:49.094 INFO:teuthology.orchestra.run.smithi073.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T23:55:49.497 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stdout a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 2024-03-08T23:55:49.497 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761... 2024-03-08T23:55:49.498 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Getting image source signatures 2024-03-08T23:55:49.498 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying blob sha256:7a785ebef89f89c1f7e1e5df571ab58da14c499794e7964e586765517f59740d 2024-03-08T23:55:49.498 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying blob sha256:f58b4670ef4403d11d1c912aa26a4dbcf73447df5302e9dc66f6a7fc0d9d1e47 2024-03-08T23:55:49.498 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Copying config sha256:a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 2024-03-08T23:55:49.498 INFO:teuthology.orchestra.run.smithi073.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-08T23:55:56.138 INFO:teuthology.orchestra.run.smithi073.stdout:ceph: stdout ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev) 2024-03-08T23:55:56.138 INFO:teuthology.orchestra.run.smithi073.stdout:Ceph version: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev) 2024-03-08T23:55:56.138 INFO:teuthology.orchestra.run.smithi073.stdout:Extracting ceph user uid/gid from container image... 2024-03-08T23:56:01.938 INFO:teuthology.orchestra.run.smithi073.stdout:stat: stdout 167 167 2024-03-08T23:56:01.939 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial keys... 2024-03-08T23:56:02.843 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQCSpetl1hwGEBAAFIOpPrX5NX9BVpmizXMGhA== 2024-03-08T23:56:04.066 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQCTpetlIxNUChAALhRvatQQMD3JjB+z8kFr9g== 2024-03-08T23:56:04.980 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph-authtool: stdout AQCUpetldJuhFRAA7+esrKbVUkNzq/gB5Vk/0w== 2024-03-08T23:56:04.980 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial monmap... 2024-03-08T23:56:05.914 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool for a [v2:172.21.15.73:3300,v1:172.21.15.73:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:setting min_mon_release = quincy 2024-03-08T23:56:05.915 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: set fsid to 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:05.916 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-08T23:56:05.916 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:56:05.916 INFO:teuthology.orchestra.run.smithi073.stdout:Creating mon... 2024-03-08T23:56:08.272 INFO:teuthology.orchestra.run.smithi073.stdout:create mon.a on 2024-03-08T23:56:08.473 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-08T23:56:08.615 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-08T23:56:08.794 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target. 2024-03-08T23:56:08.795 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target → /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target. 2024-03-08T23:56:09.042 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a 2024-03-08T23:56:09.042 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service: Unit ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service not loaded. 2024-03-08T23:56:09.200 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target.wants/ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service → /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715@.service. 2024-03-08T23:56:09.725 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T23:56:09.726 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-03-08T23:56:09.732 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T23:56:09.733 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:56:09.733 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-03-08T23:56:09.733 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to enable service . firewalld.service is not available 2024-03-08T23:56:09.733 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mon to start... 2024-03-08T23:56:09.733 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mon... 2024-03-08T23:56:10.853 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:10 smithi073 ceph-mon[31938]: from='client.? 172.21.15.73:0/508144913' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-08T23:56:10.856 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout cluster: 2024-03-08T23:56:10.862 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout id: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:10.862 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-08T23:56:10.862 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:10.862 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout services: 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.532677s) 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout data: 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout pgs: 2024-03-08T23:56:10.863 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.166+0000 7f290c14a700 1 Processor -- start 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f290c14a700 1 -- start start 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f290c14a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f290c14a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904108050 con 0x7f2904107660 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f2909ee6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f2909ee6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38194/0 (socket says 172.21.15.73:38194) 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f2909ee6700 1 -- 172.21.15.73:0/183014457 learned_addr learned my addr 172.21.15.73:0/183014457 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.167+0000 7f2909ee6700 1 -- 172.21.15.73:0/183014457 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2904108810 con 0x7f2904107660 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f2909ee6700 1 --2- 172.21.15.73:0/183014457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098a0 tx=0x7f28f4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ea7191b9e2759a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f2908ee4700 1 -- 172.21.15.73:0/183014457 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4004160 con 0x7f2904107660 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f2908ee4700 1 -- 172.21.15.73:0/183014457 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4004300 con 0x7f2904107660 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f2908ee4700 1 -- 172.21.15.73:0/183014457 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f40036a0 con 0x7f2904107660 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 -- 172.21.15.73:0/183014457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 msgr2=0x7f2904107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:10.864 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 --2- 172.21.15.73:0/183014457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f28f40098a0 tx=0x7f28f4009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 -- 172.21.15.73:0/183014457 shutdown_connections 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 --2- 172.21.15.73:0/183014457 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f2904107a80 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 -- 172.21.15.73:0/183014457 >> 172.21.15.73:0/183014457 conn(0x7f2904102c40 msgr2=0x7f2904105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 -- 172.21.15.73:0/183014457 shutdown_connections 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.168+0000 7f290c14a700 1 -- 172.21.15.73:0/183014457 wait complete. 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f290c14a700 1 Processor -- start 2024-03-08T23:56:10.865 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f290c14a700 1 -- start start 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f290c14a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f290c14a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2904108d40 con 0x7f2904107660 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f2909ee6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f2909ee6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38196/0 (socket says 172.21.15.73:38196) 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.169+0000 7f2909ee6700 1 -- 172.21.15.73:0/508144913 learned_addr learned my addr 172.21.15.73:0/508144913 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f2909ee6700 1 -- 172.21.15.73:0/508144913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f290407d0a0 con 0x7f2904107660 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f2909ee6700 1 --2- 172.21.15.73:0/508144913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f4004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4003db0 con 0x7f2904107660 2024-03-08T23:56:10.866 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28f4011630 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f28f4011950 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f290407b290 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f290407b7d0 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f28f401a460 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.170+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f28f401b780 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.171+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2904063410 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.173+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f28f4031020 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.243+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f2904194260 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f28faffd700 1 -- 172.21.15.73:0/508144913 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f28f401ba20 con 0x7f2904107660 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 msgr2=0x7f290407cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f290c14a700 1 --2- 172.21.15.73:0/508144913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f28f4000c00 tx=0x7f28f4004750 comp rx=0 tx=0).stop 2024-03-08T23:56:10.867 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 shutdown_connections 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f290c14a700 1 --2- 172.21.15.73:0/508144913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2904107660 0x7f290407cb60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.244+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 >> 172.21.15.73:0/508144913 conn(0x7f2904102c40 msgr2=0x7f29041919c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.245+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 shutdown_connections 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:10.245+0000 7f290c14a700 1 -- 172.21.15.73:0/508144913 wait complete. 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:mon is available 2024-03-08T23:56:10.868 INFO:teuthology.orchestra.run.smithi073.stdout:Assimilating anything we can from ceph.conf... 2024-03-08T23:56:11.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:11 smithi073 ceph-mon[31938]: from='client.? 172.21.15.73:0/2651148121' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-08T23:56:11.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:11 smithi073 ceph-mon[31938]: from='client.? 172.21.15.73:0/2651148121' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-08T23:56:11.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [global] 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout fsid = 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.73:3300,v1:172.21.15.73:6789] 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-08T23:56:11.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [osd] 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.257+0000 7f29a12a8700 1 Processor -- start 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.257+0000 7f29a12a8700 1 -- start start 2024-03-08T23:56:11.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.257+0000 7f29a12a8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.257+0000 7f29a12a8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c106f20 con 0x7f299c106530 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f299affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f299affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38212/0 (socket says 172.21.15.73:38212) 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f299affd700 1 -- 172.21.15.73:0/1592044879 learned_addr learned my addr 172.21.15.73:0/1592044879 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f299affd700 1 -- 172.21.15.73:0/1592044879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c107730 con 0x7f299c106530 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f299affd700 1 --2- 172.21.15.73:0/1592044879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f298c0098a0 tx=0x7f298c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=548a088592442467 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:11.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f2999ffb700 1 -- 172.21.15.73:0/1592044879 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f298c004160 con 0x7f299c106530 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f2999ffb700 1 -- 172.21.15.73:0/1592044879 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c004300 con 0x7f299c106530 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.258+0000 7f2999ffb700 1 -- 172.21.15.73:0/1592044879 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f298c0036a0 con 0x7f299c106530 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 -- 172.21.15.73:0/1592044879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 msgr2=0x7f299c106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 --2- 172.21.15.73:0/1592044879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f298c0098a0 tx=0x7f298c009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 -- 172.21.15.73:0/1592044879 shutdown_connections 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 --2- 172.21.15.73:0/1592044879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c106950 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 -- 172.21.15.73:0/1592044879 >> 172.21.15.73:0/1592044879 conn(0x7f299c101b70 msgr2=0x7f299c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 -- 172.21.15.73:0/1592044879 shutdown_connections 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 -- 172.21.15.73:0/1592044879 wait complete. 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.259+0000 7f29a12a8700 1 Processor -- start 2024-03-08T23:56:11.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f29a12a8700 1 -- start start 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f29a12a8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f29a12a8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c107c60 con 0x7f299c106530 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f299affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f299affd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38220/0 (socket says 172.21.15.73:38220) 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f299affd700 1 -- 172.21.15.73:0/2651148121 learned_addr learned my addr 172.21.15.73:0/2651148121 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f299affd700 1 -- 172.21.15.73:0/2651148121 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c19b2f0 con 0x7f299c106530 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f299affd700 1 --2- 172.21.15.73:0/2651148121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f298c003db0 con 0x7f299c106530 2024-03-08T23:56:11.925 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f298c011630 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.260+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c19b580 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.261+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f298c011950 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.261+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c19e270 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.261+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f298c01a460 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.261+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f298c01b780 con 0x7f299c106530 2024-03-08T23:56:11.926 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.261+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c063410 con 0x7f299c106530 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.264+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f298c031020 con 0x7f299c106530 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.332+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f299c19ba60 con 0x7f299c106530 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.335+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+429 (secure 0 0 0) 0x7f299c19ba60 con 0x7f299c106530 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f2983fff700 1 -- 172.21.15.73:0/2651148121 <== mon.0 v2:172.21.15.73:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f298c01a8e0 con 0x7f299c106530 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 msgr2=0x7f299c19adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f29a12a8700 1 --2- 172.21.15.73:0/2651148121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f298c000c00 tx=0x7f298c004750 comp rx=0 tx=0).stop 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 shutdown_connections 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f29a12a8700 1 --2- 172.21.15.73:0/2651148121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106530 0x7f299c19adb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.336+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 >> 172.21.15.73:0/2651148121 conn(0x7f299c101b70 msgr2=0x7f299c079fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:11.927 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.337+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 shutdown_connections 2024-03-08T23:56:11.928 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:11.337+0000 7f29a12a8700 1 -- 172.21.15.73:0/2651148121 wait complete. 2024-03-08T23:56:11.928 INFO:teuthology.orchestra.run.smithi073.stdout:Generating new minimal ceph.conf... 2024-03-08T23:56:12.899 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.304+0000 7f620297d700 1 Processor -- start 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.304+0000 7f620297d700 1 -- start start 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f620297d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f620297d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc07b7f0 con 0x7f61fc07ce00 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61fbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61fbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38222/0 (socket says 172.21.15.73:38222) 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61fbfff700 1 -- 172.21.15.73:0/3656733620 learned_addr learned my addr 172.21.15.73:0/3656733620 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61fbfff700 1 -- 172.21.15.73:0/3656733620 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc07b970 con 0x7f61fc07ce00 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61fbfff700 1 --2- 172.21.15.73:0/3656733620 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f61f00098a0 tx=0x7f61f0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ace6861bf145d6a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:12.900 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61faffd700 1 -- 172.21.15.73:0/3656733620 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61f0004160 con 0x7f61fc07ce00 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61faffd700 1 -- 172.21.15.73:0/3656733620 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61f0004300 con 0x7f61fc07ce00 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.305+0000 7f61faffd700 1 -- 172.21.15.73:0/3656733620 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61f0011b50 con 0x7f61fc07ce00 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 -- 172.21.15.73:0/3656733620 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 msgr2=0x7f61fc07b2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 --2- 172.21.15.73:0/3656733620 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f61f00098a0 tx=0x7f61f0009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 -- 172.21.15.73:0/3656733620 shutdown_connections 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 --2- 172.21.15.73:0/3656733620 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc07b2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 -- 172.21.15.73:0/3656733620 >> 172.21.15.73:0/3656733620 conn(0x7f61fc102c40 msgr2=0x7f61fc105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 -- 172.21.15.73:0/3656733620 shutdown_connections 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 -- 172.21.15.73:0/3656733620 wait complete. 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.306+0000 7f620297d700 1 Processor -- start 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f620297d700 1 -- start start 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f620297d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:12.901 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f620297d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61fc10dc50 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61fbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61fbfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38224/0 (socket says 172.21.15.73:38224) 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61fbfff700 1 -- 172.21.15.73:0/98439158 learned_addr learned my addr 172.21.15.73:0/98439158 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61fbfff700 1 -- 172.21.15.73:0/98439158 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61fc19dd40 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61fbfff700 1 --2- 172.21.15.73:0/98439158 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009fb0 tx=0x7f61f0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61f00038b0 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f61f0011e50 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.307+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61fc19dfd0 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.308+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f61f0019420 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.308+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61fc1a0cc0 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.308+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f61f0019b30 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.308+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f61f000bb10 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.309+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61fc063410 con 0x7f61fc07ce00 2024-03-08T23:56:12.902 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.311+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f61f0032070 con 0x7f61fc07ce00 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.379+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f61fc1a0e90 con 0x7f61fc07ce00 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.379+0000 7f61f97fa700 1 -- 172.21.15.73:0/98439158 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f61f0003a50 con 0x7f61fc07ce00 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 msgr2=0x7f61fc19d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 --2- 172.21.15.73:0/98439158 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f61f0009fb0 tx=0x7f61f0011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 shutdown_connections 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 --2- 172.21.15.73:0/98439158 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61fc07ce00 0x7f61fc19d800 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 >> 172.21.15.73:0/98439158 conn(0x7f61fc102c40 msgr2=0x7f61fc1037d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 shutdown_connections 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:12.380+0000 7f620297d700 1 -- 172.21.15.73:0/98439158 wait complete. 2024-03-08T23:56:12.903 INFO:teuthology.orchestra.run.smithi073.stdout:Restarting the monitor... 2024-03-08T23:56:12.919 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:12 smithi073 ceph-mon[31938]: from='client.? 172.21.15.73:0/98439158' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:56:13.178 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:12 smithi073 systemd[1]: Stopping Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-08T23:56:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a[31934]: 2024-03-08T23:56:13.182+0000 7f1b108ca700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-08T23:56:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a[31934]: 2024-03-08T23:56:13.182+0000 7f1b108ca700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-08T23:56:13.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 podman[32185]: 2024-03-08 23:56:13.356616727 +0000 UTC m=+0.293616849 container died ad0086c00400cf792bb69c9180e95ea51f8e32c9d168aa65fc8d44d015bbb974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307) 2024-03-08T23:56:13.937 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 podman[32185]: 2024-03-08 23:56:13.757603743 +0000 UTC m=+0.694603874 container cleanup ad0086c00400cf792bb69c9180e95ea51f8e32c9d168aa65fc8d44d015bbb974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:56:13.937 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 bash[32185]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a 2024-03-08T23:56:14.276 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:13 smithi073 podman[32199]: 2024-03-08 23:56:13.94192094 +0000 UTC m=+0.585824480 container remove ad0086c00400cf792bb69c9180e95ea51f8e32c9d168aa65fc8d44d015bbb974 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-08T23:56:14.276 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-08T23:56:14.276 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 systemd[1]: Stopped Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:56:14.276 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 systemd[1]: Starting Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-08T23:56:14.531 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 podman[32290]: 2024-03-08 23:56:14.182692746 +0000 UTC m=+0.017477633 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:56:14.531 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 podman[32290]: 2024-03-08 23:56:14.318129972 +0000 UTC m=+0.152914886 container create 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-08T23:56:14.531 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 podman[32290]: 2024-03-08 23:56:14.503634599 +0000 UTC m=+0.338419468 container init 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-08T23:56:14.531 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 podman[32290]: 2024-03-08 23:56:14.506399504 +0000 UTC m=+0.341184378 container start 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-08T23:56:14.532 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: set uid:gid to 167:167 (ceph:ceph) 2024-03-08T23:56:14.532 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev), process ceph-mon, pid 2 2024-03-08T23:56:14.532 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: pidfile_write: ignore empty --pid-file 2024-03-08T23:56:14.532 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: load: jerasure load: lrc 2024-03-08T23:56:14.586 INFO:teuthology.orchestra.run.smithi073.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-08T23:56:14.811 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: RocksDB version: 7.9.2 2024-03-08T23:56:14.811 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Git sha 0 2024-03-08T23:56:14.811 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Compile date 2024-03-08 15:06:51 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: DB SUMMARY 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: DB Session ID: VU6DLPO5L40A0UWUXIBW 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: CURRENT file: CURRENT 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: IDENTITY file: IDENTITY 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 99138 ; 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.error_if_exists: 0 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.create_if_missing: 0 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.paranoid_checks: 1 2024-03-08T23:56:14.812 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.env: 0x55dc4100ab20 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.fs: PosixFileSystem 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.info_log: 0x55dc4376f8e0 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_file_opening_threads: 16 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.statistics: (nil) 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.use_fsync: 0 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_log_file_size: 0 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-08T23:56:14.813 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.keep_log_file_num: 1000 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.recycle_log_file_num: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_fallocate: 1 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_mmap_reads: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_mmap_writes: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.use_direct_reads: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.create_missing_column_families: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.db_log_dir: 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.wal_dir: 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-08T23:56:14.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.advise_random_on_open: 1 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.db_write_buffer_size: 0 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.write_buffer_manager: 0x55dc43743400 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.rate_limiter: (nil) 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.wal_recovery_mode: 2 2024-03-08T23:56:14.815 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enable_thread_tracking: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enable_pipelined_write: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.unordered_write: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.row_cache: None 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.wal_filter: None 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_ingest_behind: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.two_write_queues: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.manual_wal_flush: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.wal_compression: 0 2024-03-08T23:56:14.816 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.atomic_flush: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.log_readahead_size: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.best_efforts_recovery: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.allow_data_in_errors: 0 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.db_host_id: __hostname__ 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_background_jobs: 2 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_background_compactions: -1 2024-03-08T23:56:14.817 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_subcompactions: 1 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_total_wal_size: 0 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-08T23:56:14.819 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_open_files: -1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bytes_per_sync: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_readahead_size: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_background_flushes: -1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Compression algorithms supported: 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kZSTD supported: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kXpressCompression supported: 0 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kLZ4HCCompression supported: 1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kZlibCompression supported: 1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kSnappyCompression supported: 1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kLZ4Compression supported: 1 2024-03-08T23:56:14.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: kBZip2Compression supported: 0 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.merge_operator: 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_filter: None 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_filter_factory: None 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.sst_partitioner_factory: None 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55dc4376f9e0) 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout: cache_index_and_filter_blocks: 1 2024-03-08T23:56:14.821 INFO:journalctl@ceph.mon.a.smithi073.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: pin_top_level_index_and_filter: 1 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: index_type: 0 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: data_block_index_type: 0 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: index_shortening: 1 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: checksum: 4 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: no_block_cache: 0 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_cache: 0x55dc43775090 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_cache_name: BinnedLRUCache 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_cache_options: 2024-03-08T23:56:14.822 INFO:journalctl@ceph.mon.a.smithi073.stdout: capacity : 536870912 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: num_shard_bits : 4 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: strict_capacity_limit : 0 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: high_pri_pool_ratio: 0.000 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_cache_compressed: (nil) 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: persistent_cache: (nil) 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_size: 4096 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_size_deviation: 10 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_restart_interval: 16 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: index_block_restart_interval: 1 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: metadata_block_size: 4096 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: partition_filters: 0 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: use_delta_encoding: 1 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: filter_policy: bloomfilter 2024-03-08T23:56:14.823 INFO:journalctl@ceph.mon.a.smithi073.stdout: whole_key_filtering: 1 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: verify_compression: 0 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: read_amp_bytes_per_bit: 0 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: format_version: 5 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: enable_index_compression: 1 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: block_align: 0 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: max_auto_readahead_size: 262144 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: prepopulate_block_cache: 0 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: initial_auto_readahead_size: 8192 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout: num_file_reads_for_auto_readahead: 2 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.write_buffer_size: 33554432 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_write_buffer_number: 2 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression: NoCompression 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression: Disabled 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.prefix_extractor: nullptr 2024-03-08T23:56:14.824 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.num_levels: 7 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-08T23:56:14.825 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.level: 32767 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.strategy: 0 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.enabled: false 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.target_file_size_base: 67108864 2024-03-08T23:56:14.826 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.arena_block_size: 1048576 2024-03-08T23:56:14.827 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.disable_auto_compactions: 0 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.table_properties_collectors: 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.inplace_update_support: 0 2024-03-08T23:56:14.828 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.bloom_locality: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.max_successive_merges: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.paranoid_file_checks: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.force_consistency_checks: 1 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.report_bg_io_stats: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.ttl: 2592000 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-08T23:56:14.829 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enable_blob_files: false 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.min_blob_size: 0 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_file_size: 268435456 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.blob_file_starting_level: 0 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 534a6031-8b31-4aa5-802b-309bae86099a 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942174534065, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-08T23:56:14.830 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942174535038, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 94412, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 314, "table_properties": {"data_size": 92412, "index_size": 282, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13640, "raw_average_key_size": 49, "raw_value_size": 84949, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 273, "num_filter_entries": 273, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709942174, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "534a6031-8b31-4aa5-802b-309bae86099a", "db_session_id": "VU6DLPO5L40A0UWUXIBW", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942174535118, "job": 1, "event": "recovery_finished"} 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x55dc43868000 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: DB pointer 0x55dc43852000 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: ** DB Stats ** 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T23:56:14.831 INFO:journalctl@ceph.mon.a.smithi073.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: L0 2/0 94.03 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Sum 2/0 94.03 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: ** Compaction Stats [default] ** 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-08T23:56:14.832 INFO:journalctl@ceph.mon.a.smithi073.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 142.7 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Cumulative compaction: 0.00 GB write, 31.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Interval compaction: 0.00 GB write, 31.29 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Block cache BinnedLRUCache@0x55dc43775090#2 capacity: 512.00 MB usage: 1.36 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 8e-06 secs_since: 0 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.47 KB,8.9407e-05%) Misc(1,0.00 KB,0%) 2024-03-08T23:56:14.833 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: starting mon.a rank 0 at public addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] at bind addrs [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???) e1 preinit fsid 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).mds e1 new map 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).mds e1 print_map 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: e1 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: legacy client fscid: -1 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout: No filesystems configured 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-08T23:56:14.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 bash[32290]: 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 systemd[1]: Started Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: monmap e1: 1 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: fsmap 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: osdmap e1: 0 total, 0 up, 0 in 2024-03-08T23:56:14.835 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:14 smithi073 ceph-mon[32309]: mgrmap e1: no daemons active 2024-03-08T23:56:15.593 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.068+0000 7f812f7d6700 1 Processor -- start 2024-03-08T23:56:15.593 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812f7d6700 1 -- start start 2024-03-08T23:56:15.593 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812f7d6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:15.593 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812f7d6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812807adc0 con 0x7f8128108180 2024-03-08T23:56:15.593 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812d572700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812d572700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38238/0 (socket says 172.21.15.73:38238) 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.069+0000 7f812d572700 1 -- 172.21.15.73:0/1386948032 learned_addr learned my addr 172.21.15.73:0/1386948032 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812d572700 1 -- 172.21.15.73:0/1386948032 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812810aab0 con 0x7f8128108180 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812d572700 1 --2- 172.21.15.73:0/1386948032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f81240098a0 tx=0x7f8124009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67ed9a5020a73df9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f811ffff700 1 -- 172.21.15.73:0/1386948032 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8124004160 con 0x7f8128108180 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f811ffff700 1 -- 172.21.15.73:0/1386948032 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8124004300 con 0x7f8128108180 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f811ffff700 1 -- 172.21.15.73:0/1386948032 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8124011b50 con 0x7f8128108180 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812f7d6700 1 -- 172.21.15.73:0/1386948032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 msgr2=0x7f812810a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812f7d6700 1 --2- 172.21.15.73:0/1386948032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f81240098a0 tx=0x7f8124009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812f7d6700 1 -- 172.21.15.73:0/1386948032 shutdown_connections 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812f7d6700 1 --2- 172.21.15.73:0/1386948032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812810a570 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.070+0000 7f812f7d6700 1 -- 172.21.15.73:0/1386948032 >> 172.21.15.73:0/1386948032 conn(0x7f8128101b70 msgr2=0x7f8128103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:15.594 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 -- 172.21.15.73:0/1386948032 shutdown_connections 2024-03-08T23:56:15.595 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 -- 172.21.15.73:0/1386948032 wait complete. 2024-03-08T23:56:15.597 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 Processor -- start 2024-03-08T23:56:15.597 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 -- start start 2024-03-08T23:56:15.597 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812f7d6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f812810ba10 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812d572700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812d572700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38252/0 (socket says 172.21.15.73:38252) 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.071+0000 7f812d572700 1 -- 172.21.15.73:0/3836864708 learned_addr learned my addr 172.21.15.73:0/3836864708 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f812d572700 1 -- 172.21.15.73:0/3836864708 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f812819bb30 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f812d572700 1 --2- 172.21.15.73:0/3836864708 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8124009fb0 tx=0x7f8124011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81240038b0 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f812819bdc0 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8124011e50 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8124019420 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f812819eab0 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8124019ad0 con 0x7f8128108180 2024-03-08T23:56:15.598 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.072+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8124024730 con 0x7f8128108180 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.073+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8128063410 con 0x7f8128108180 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.075+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f8124043e50 con 0x7f8128108180 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.142+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f812819ec80 con 0x7f8128108180 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.143+0000 7f811e7fc700 1 -- 172.21.15.73:0/3836864708 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f8124032030 con 0x7f8128108180 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 msgr2=0x7f812819b5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 --2- 172.21.15.73:0/3836864708 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8124009fb0 tx=0x7f8124011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 shutdown_connections 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 --2- 172.21.15.73:0/3836864708 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8128108180 0x7f812819b5f0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 >> 172.21.15.73:0/3836864708 conn(0x7f8128101b70 msgr2=0x7f81281026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 shutdown_connections 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:15.144+0000 7f812f7d6700 1 -- 172.21.15.73:0/3836864708 wait complete. 2024-03-08T23:56:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-08T23:56:15.614 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:56:15.614 INFO:teuthology.orchestra.run.smithi073.stdout:Creating mgr... 2024-03-08T23:56:15.615 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-08T23:56:15.615 INFO:teuthology.orchestra.run.smithi073.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-08T23:56:15.869 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x 2024-03-08T23:56:15.870 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Failed to reset failed state of unit ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service: Unit ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service not loaded. 2024-03-08T23:56:16.019 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715.target.wants/ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-44f886a8-dda7-11ee-95c6-87774f69a715@.service. 2024-03-08T23:56:16.140 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 systemd[1]: Starting Ceph mgr.x for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-08T23:56:16.421 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:16 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3836864708' entity='client.admin' 2024-03-08T23:56:16.421 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 podman[32524]: 2024-03-08 23:56:16.159687589 +0000 UTC m=+0.018575710 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 2024-03-08T23:56:16.421 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 podman[32524]: 2024-03-08 23:56:16.262829582 +0000 UTC m=+0.121717688 container create 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-08T23:56:16.566 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T23:56:16.567 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-03-08T23:56:16.574 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T23:56:16.574 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:56:16.574 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-03-08T23:56:16.574 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to enable service . firewalld.service is not available 2024-03-08T23:56:16.581 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T23:56:16.582 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr to start... 2024-03-08T23:56:16.590 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr... 2024-03-08T23:56:16.728 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 podman[32524]: 2024-03-08 23:56:16.473849809 +0000 UTC m=+0.332737915 container init 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-08T23:56:16.728 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 podman[32524]: 2024-03-08 23:56:16.476539591 +0000 UTC m=+0.335427698 container start 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-08T23:56:16.728 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 bash[32524]: 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 2024-03-08T23:56:16.728 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:16 smithi073 systemd[1]: Started Ceph mgr.x for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:56:17.321 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:17 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2363831320' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T23:56:17.321 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:17.033+0000 7f638dbc5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:56:17.321 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:17.129+0000 7f638dbc5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:56:17.755 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T23:56:17.756 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T23:56:17.757 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T23:56:17.765 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T23:56:17.766 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T23:56:09.713778+0000", 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.060+0000 7fa597b2f700 1 Processor -- start 2024-03-08T23:56:17.767 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.060+0000 7fa597b2f700 1 -- start start 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.060+0000 7fa597b2f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.060+0000 7fa597b2f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa590106f40 con 0x7fa590106550 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5958cb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5958cb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38278/0 (socket says 172.21.15.73:38278) 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5958cb700 1 -- 172.21.15.73:0/3557013584 learned_addr learned my addr 172.21.15.73:0/3557013584 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5958cb700 1 -- 172.21.15.73:0/3557013584 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa590107750 con 0x7fa590106550 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5958cb700 1 --2- 172.21.15.73:0/3557013584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa5800098a0 tx=0x7fa580009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=278442db1a721023 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5948c9700 1 -- 172.21.15.73:0/3557013584 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa580004160 con 0x7fa590106550 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5948c9700 1 -- 172.21.15.73:0/3557013584 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa580004300 con 0x7fa590106550 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.061+0000 7fa5948c9700 1 -- 172.21.15.73:0/3557013584 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa580011b50 con 0x7fa590106550 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 -- 172.21.15.73:0/3557013584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 msgr2=0x7fa590106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 --2- 172.21.15.73:0/3557013584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa5800098a0 tx=0x7fa580009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 -- 172.21.15.73:0/3557013584 shutdown_connections 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 --2- 172.21.15.73:0/3557013584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa590106970 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:17.768 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 -- 172.21.15.73:0/3557013584 >> 172.21.15.73:0/3557013584 conn(0x7fa590101b70 msgr2=0x7fa590103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 -- 172.21.15.73:0/3557013584 shutdown_connections 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.062+0000 7fa597b2f700 1 -- 172.21.15.73:0/3557013584 wait complete. 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa597b2f700 1 Processor -- start 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa597b2f700 1 -- start start 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa597b2f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa597b2f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59010ab00 con 0x7fa590106550 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa5958cb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa5958cb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38292/0 (socket says 172.21.15.73:38292) 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa5958cb700 1 -- 172.21.15.73:0/2363831320 learned_addr learned my addr 172.21.15.73:0/2363831320 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa5958cb700 1 -- 172.21.15.73:0/2363831320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa59019bb90 con 0x7fa590106550 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.063+0000 7fa5958cb700 1 --2- 172.21.15.73:0/2363831320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa580009fb0 tx=0x7fa580011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa5800038b0 con 0x7fa590106550 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa59019be20 con 0x7fa590106550 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa580011e50 con 0x7fa590106550 2024-03-08T23:56:17.769 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fa580019550 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa59019eb10 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7fa580019be0 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fa580024730 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.064+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa590063410 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.067+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fa580003a50 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.140+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fa5900008d0 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.141+0000 7fa586ffd700 1 -- 172.21.15.73:0/2363831320 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fa5800249d0 con 0x7fa590106550 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.141+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 msgr2=0x7fa59019b650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.141+0000 7fa597b2f700 1 --2- 172.21.15.73:0/2363831320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa580009fb0 tx=0x7fa580011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.141+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 shutdown_connections 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.142+0000 7fa597b2f700 1 --2- 172.21.15.73:0/2363831320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa590106550 0x7fa59019b650 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:17.770 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.142+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 >> 172.21.15.73:0/2363831320 conn(0x7fa590101b70 msgr2=0x7fa590102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:17.771 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.142+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 shutdown_connections 2024-03-08T23:56:17.771 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:17.142+0000 7fa597b2f700 1 -- 172.21.15.73:0/2363831320 wait complete. 2024-03-08T23:56:17.771 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (1/15)... 2024-03-08T23:56:18.011 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:17.787+0000 7f638dbc5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:56:18.011 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:17.876+0000 7f638dbc5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:56:18.011 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.014+0000 7f638dbc5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:56:18.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.109+0000 7f638dbc5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:56:18.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.190+0000 7f638dbc5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:56:18.733 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.450+0000 7f638dbc5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:56:18.733 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.558+0000 7f638dbc5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:56:18.733 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.643+0000 7f638dbc5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:56:18.733 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.737+0000 7f638dbc5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:56:19.152 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:18.911+0000 7f638dbc5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:56:20.120 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:19.872+0000 7f638dbc5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:56:20.120 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:20.002+0000 7f638dbc5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:56:20.380 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:20.353+0000 7f638dbc5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:56:20.649 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2859846659' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T23:56:20.649 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:20.453+0000 7f638dbc5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:56:20.902 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:20.652+0000 7f638dbc5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:56:20.902 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:20.838+0000 7f638dbc5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T23:56:20.954 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.955 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T23:56:20.956 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T23:56:20.957 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T23:56:09.713778+0000", 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bca7e0700 1 Processor -- start 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bca7e0700 1 -- start start 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bca7e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:20.958 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bca7e0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc407b8c0 con 0x7f9bc407ce40 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bc3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bc3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44530/0 (socket says 172.21.15.73:44530) 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.226+0000 7f9bc3fff700 1 -- 172.21.15.73:0/3247405183 learned_addr learned my addr 172.21.15.73:0/3247405183 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.227+0000 7f9bc3fff700 1 -- 172.21.15.73:0/3247405183 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc407ba40 con 0x7f9bc407ce40 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.227+0000 7f9bc3fff700 1 --2- 172.21.15.73:0/3247405183 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098a0 tx=0x7f9bb4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a70ccb35d5d7059 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.227+0000 7f9bc2ffd700 1 -- 172.21.15.73:0/3247405183 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bb4004160 con 0x7f9bc407ce40 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.227+0000 7f9bc2ffd700 1 -- 172.21.15.73:0/3247405183 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb4004300 con 0x7f9bc407ce40 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.227+0000 7f9bc2ffd700 1 -- 172.21.15.73:0/3247405183 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bb4011b50 con 0x7f9bc407ce40 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- 172.21.15.73:0/3247405183 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 msgr2=0x7f9bc407b2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 --2- 172.21.15.73:0/3247405183 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f9bb40098a0 tx=0x7f9bb4009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- 172.21.15.73:0/3247405183 shutdown_connections 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 --2- 172.21.15.73:0/3247405183 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc407b2f0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- 172.21.15.73:0/3247405183 >> 172.21.15.73:0/3247405183 conn(0x7f9bc4102c80 msgr2=0x7f9bc41050a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:20.959 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- 172.21.15.73:0/3247405183 shutdown_connections 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- 172.21.15.73:0/3247405183 wait complete. 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 Processor -- start 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.228+0000 7f9bca7e0700 1 -- start start 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bca7e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bca7e0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bc410dbf0 con 0x7f9bc407ce40 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc3fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44536/0 (socket says 172.21.15.73:44536) 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc3fff700 1 -- 172.21.15.73:0/2859846659 learned_addr learned my addr 172.21.15.73:0/2859846659 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc3fff700 1 -- 172.21.15.73:0/2859846659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bc419dce0 con 0x7f9bc407ce40 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc3fff700 1 --2- 172.21.15.73:0/2859846659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4011670 tx=0x7f9bb4011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.229+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bb40039c0 con 0x7f9bc407ce40 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bc419df70 con 0x7f9bc407ce40 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9bb4011e50 con 0x7f9bc407ce40 2024-03-08T23:56:20.960 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9bb401a5c0 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bc41a0c60 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f9bb401a8c0 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.230+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9bb40193c0 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.231+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bc4063410 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.233+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7f9bb402de10 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.332+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f9bc41a0e30 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.333+0000 7f9bc17fa700 1 -- 172.21.15.73:0/2859846659 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f9bb4032030 con 0x7f9bc407ce40 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 msgr2=0x7f9bc419d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 --2- 172.21.15.73:0/2859846659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9bb4011670 tx=0x7f9bb4011b60 comp rx=0 tx=0).stop 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 shutdown_connections 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 --2- 172.21.15.73:0/2859846659 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bc407ce40 0x7f9bc419d7a0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 >> 172.21.15.73:0/2859846659 conn(0x7f9bc4102c80 msgr2=0x7f9bc4103790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 shutdown_connections 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:20.334+0000 7f9bca7e0700 1 -- 172.21.15.73:0/2859846659 wait complete. 2024-03-08T23:56:20.961 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (2/15)... 2024-03-08T23:56:21.375 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:21.126+0000 7f638dbc5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:56:21.376 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:21.379+0000 7f638dbc5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:56:21.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:21.615+0000 7f638dbc5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:56:22.902 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:22.408+0000 7f638dbc5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:56:23.285 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:22.973+0000 7f638dbc5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:56:23.285 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:23.074+0000 7f638dbc5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:56:23.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:23.296+0000 7f638dbc5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: Activating manager daemon x 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: mgrmap e2: x(active, starting, since 0.00291841s) 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: Manager daemon x is now available 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:56:23.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' 2024-03-08T23:56:23.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' 2024-03-08T23:56:23.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:23 smithi073 ceph-mon[32309]: from='mgr.14100 172.21.15.73:0/1174683094' entity='mgr.x' 2024-03-08T23:56:24.061 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:24.061 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:24.061 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-08T23:56:24.061 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:24.062 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T23:56:24.063 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T23:56:24.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-08T23:56:24.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T23:56:09.713778+0000", 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:24.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.392+0000 7fb4b5324700 1 Processor -- start 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.392+0000 7fb4b5324700 1 -- start start 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4b5324700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4b5324700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b0109680 con 0x7fb4b0108c90 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4aeffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4aeffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44608/0 (socket says 172.21.15.73:44608) 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4aeffd700 1 -- 172.21.15.73:0/3758791462 learned_addr learned my addr 172.21.15.73:0/3758791462 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4aeffd700 1 -- 172.21.15.73:0/3758791462 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b0109800 con 0x7fb4b0108c90 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4aeffd700 1 --2- 172.21.15.73:0/3758791462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00098a0 tx=0x7fb4a0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74a49805f160c3a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4ae7fc700 1 -- 172.21.15.73:0/3758791462 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4a0004160 con 0x7fb4b0108c90 2024-03-08T23:56:24.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4ae7fc700 1 -- 172.21.15.73:0/3758791462 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4a0004300 con 0x7fb4b0108c90 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.393+0000 7fb4ae7fc700 1 -- 172.21.15.73:0/3758791462 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4a0011b50 con 0x7fb4b0108c90 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 -- 172.21.15.73:0/3758791462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 msgr2=0x7fb4b01090b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 --2- 172.21.15.73:0/3758791462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb4a00098a0 tx=0x7fb4a0009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 -- 172.21.15.73:0/3758791462 shutdown_connections 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 --2- 172.21.15.73:0/3758791462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b01090b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 -- 172.21.15.73:0/3758791462 >> 172.21.15.73:0/3758791462 conn(0x7fb4b01012d0 msgr2=0x7fb4b01036f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 -- 172.21.15.73:0/3758791462 shutdown_connections 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.394+0000 7fb4b5324700 1 -- 172.21.15.73:0/3758791462 wait complete. 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4b5324700 1 Processor -- start 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4b5324700 1 -- start start 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4b5324700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:24.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4b5324700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b01064f0 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4aeffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4aeffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44618/0 (socket says 172.21.15.73:44618) 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4aeffd700 1 -- 172.21.15.73:0/1473235426 learned_addr learned my addr 172.21.15.73:0/1473235426 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.395+0000 7fb4aeffd700 1 -- 172.21.15.73:0/1473235426 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b019bb40 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4aeffd700 1 --2- 172.21.15.73:0/1473235426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009fb0 tx=0x7fb4a0011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4a00038b0 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb4a0011e50 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b019be30 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb4a0019550 con 0x7fb4b0108c90 2024-03-08T23:56:24.077 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.396+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b019eac0 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.397+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 2) v1 ==== 45862+0+0 (secure 0 0 0) 0x7fb4a0019800 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.397+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b0063410 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.397+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fb4a004cd30 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.399+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73265 (secure 0 0 0) 0x7fb4a0019e20 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.468+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fb4b019f240 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4acff9700 1 -- 172.21.15.73:0/1473235426 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7fb4a0003ac0 con 0x7fb4b0108c90 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 msgr2=0x7fb4b019b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 --2- 172.21.15.73:0/1473235426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fb4a0009fb0 tx=0x7fb4a0011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 shutdown_connections 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 --2- 172.21.15.73:0/1473235426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb4b0108c90 0x7fb4b019b600 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 >> 172.21.15.73:0/1473235426 conn(0x7fb4b01012d0 msgr2=0x7fb4b0101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:24.078 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 shutdown_connections 2024-03-08T23:56:24.079 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:23.469+0000 7fb4b5324700 1 -- 172.21.15.73:0/1473235426 wait complete. 2024-03-08T23:56:24.079 INFO:teuthology.orchestra.run.smithi073.stdout:mgr not available, waiting (3/15)... 2024-03-08T23:56:24.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:24 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1473235426' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T23:56:24.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:24 smithi073 ceph-mon[32309]: mgrmap e3: x(active, since 1.00621s) 2024-03-08T23:56:26.484 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:26 smithi073 ceph-mon[32309]: mgrmap e4: x(active, since 2s) 2024-03-08T23:56:27.521 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/875892163' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "health": { 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.526 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 0 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "a" 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "quorum_age": 12, 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:27.527 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-08T23:56:27.528 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-08T23:56:27.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "restful" 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ], 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "modified": "2024-03-08T23:56:09.713778+0000", 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-08T23:56:27.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout }, 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f408a368700 1 Processor -- start 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f408a368700 1 -- start start 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f408a368700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f408a368700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4084107ff0 con 0x7f4084107600 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f4083fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f4083fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44626/0 (socket says 172.21.15.73:44626) 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f4083fff700 1 -- 172.21.15.73:0/3019734791 learned_addr learned my addr 172.21.15.73:0/3019734791 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.510+0000 7f4083fff700 1 -- 172.21.15.73:0/3019734791 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40841087b0 con 0x7f4084107600 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f4083fff700 1 --2- 172.21.15.73:0/3019734791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f40740098a0 tx=0x7f4074009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d68f286f47162b4e server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f4082ffd700 1 -- 172.21.15.73:0/3019734791 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4074004160 con 0x7f4084107600 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f4082ffd700 1 -- 172.21.15.73:0/3019734791 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4074004300 con 0x7f4084107600 2024-03-08T23:56:27.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f4082ffd700 1 -- 172.21.15.73:0/3019734791 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4074011b50 con 0x7f4084107600 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 -- 172.21.15.73:0/3019734791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 msgr2=0x7f4084107a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 --2- 172.21.15.73:0/3019734791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f40740098a0 tx=0x7f4074009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 -- 172.21.15.73:0/3019734791 shutdown_connections 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 --2- 172.21.15.73:0/3019734791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f4084107a20 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 -- 172.21.15.73:0/3019734791 >> 172.21.15.73:0/3019734791 conn(0x7f4084102be0 msgr2=0x7f4084105000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 -- 172.21.15.73:0/3019734791 shutdown_connections 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.511+0000 7f408a368700 1 -- 172.21.15.73:0/3019734791 wait complete. 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f408a368700 1 Processor -- start 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f408a368700 1 -- start start 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f408a368700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f408a368700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f408410bb60 con 0x7f4084107600 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f4083fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f4083fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44632/0 (socket says 172.21.15.73:44632) 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.512+0000 7f4083fff700 1 -- 172.21.15.73:0/875892163 learned_addr learned my addr 172.21.15.73:0/875892163 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:27.540 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f4083fff700 1 -- 172.21.15.73:0/875892163 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f408407d0a0 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f4083fff700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4074011670 tx=0x7f4074011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f40740039c0 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4074011e50 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f408407b290 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f407401a5c0 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.513+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f408407b7d0 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.514+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f407401a870 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.514+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4084063410 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.514+0000 7f40817fa700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 0x7f406c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.514+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f407404d3e0 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.514+0000 7f40837fe700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 0x7f406c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.515+0000 7f40837fe700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 0x7f406c03b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4078006fd0 tx=0x7f4078006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.519+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4074019c70 con 0x7f4084107600 2024-03-08T23:56:27.541 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.791+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f408407ba50 con 0x7f4084107600 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.792+0000 7f40817fa700 1 -- 172.21.15.73:0/875892163 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f407402cc00 con 0x7f4084107600 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 msgr2=0x7f406c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 0x7f406c03b390 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f4078006fd0 tx=0x7f4078006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 msgr2=0x7f408407cb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f4074011670 tx=0x7f4074011b60 comp rx=0 tx=0).stop 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 shutdown_connections 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f406c038ed0 0x7f406c03b390 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 --2- 172.21.15.73:0/875892163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4084107600 0x7f408407cb60 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 >> 172.21.15.73:0/875892163 conn(0x7f4084102be0 msgr2=0x7f4084103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 shutdown_connections 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:26.794+0000 7f408a368700 1 -- 172.21.15.73:0/875892163 wait complete. 2024-03-08T23:56:27.542 INFO:teuthology.orchestra.run.smithi073.stdout:mgr is available 2024-03-08T23:56:28.443 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:28 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/299955406' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [global] 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout fsid = 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-08T23:56:28.739 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout [osd] 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feacd28d700 1 Processor -- start 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feacd28d700 1 -- start start 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feacd28d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feacd28d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac807ae40 con 0x7feac8105f10 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feac6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feac6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44638/0 (socket says 172.21.15.73:44638) 2024-03-08T23:56:28.740 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feac6ffd700 1 -- 172.21.15.73:0/1412519780 learned_addr learned my addr 172.21.15.73:0/1412519780 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feac6ffd700 1 -- 172.21.15.73:0/1412519780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac8106870 con 0x7feac8105f10 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.961+0000 7feac6ffd700 1 --2- 172.21.15.73:0/1412519780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feab80098a0 tx=0x7feab8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb747512479622ea server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feac5ffb700 1 -- 172.21.15.73:0/1412519780 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8004160 con 0x7feac8105f10 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feac5ffb700 1 -- 172.21.15.73:0/1412519780 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feab8004300 con 0x7feac8105f10 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feac5ffb700 1 -- 172.21.15.73:0/1412519780 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8011b50 con 0x7feac8105f10 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 -- 172.21.15.73:0/1412519780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 msgr2=0x7feac8106330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 --2- 172.21.15.73:0/1412519780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7feab80098a0 tx=0x7feab8009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:28.741 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 -- 172.21.15.73:0/1412519780 shutdown_connections 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 --2- 172.21.15.73:0/1412519780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac8106330 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 -- 172.21.15.73:0/1412519780 >> 172.21.15.73:0/1412519780 conn(0x7feac8101b70 msgr2=0x7feac8103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 -- 172.21.15.73:0/1412519780 shutdown_connections 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.962+0000 7feacd28d700 1 -- 172.21.15.73:0/1412519780 wait complete. 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feacd28d700 1 Processor -- start 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feacd28d700 1 -- start start 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feacd28d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feacd28d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feac8107900 con 0x7feac8105f10 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feac6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feac6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44640/0 (socket says 172.21.15.73:44640) 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.963+0000 7feac6ffd700 1 -- 172.21.15.73:0/299955406 learned_addr learned my addr 172.21.15.73:0/299955406 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feac6ffd700 1 -- 172.21.15.73:0/299955406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feac819dd40 con 0x7feac8105f10 2024-03-08T23:56:28.742 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feac6ffd700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feab8011b60 tx=0x7feab8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8003970 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feab8003b10 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feac819dfd0 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7feab8024610 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.964+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feac81a0cc0 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7feab8024910 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feaaffff700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 0x7feab003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7feab8011e00 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feac8063410 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feac67fc700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 0x7feab003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.965+0000 7feac67fc700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 0x7feab003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feabc006fd0 tx=0x7feabc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:27.970+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feab800ba00 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.163+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7feac802da40 con 0x7feac8105f10 2024-03-08T23:56:28.743 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.164+0000 7feaaffff700 1 -- 172.21.15.73:0/299955406 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7feab8032020 con 0x7feac8105f10 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 msgr2=0x7feab003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 0x7feab003b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7feabc006fd0 tx=0x7feabc006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 msgr2=0x7feac819d800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7feab8011b60 tx=0x7feab8011c40 comp rx=0 tx=0).stop 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 shutdown_connections 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7feab0038ed0 0x7feab003b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 --2- 172.21.15.73:0/299955406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feac8105f10 0x7feac819d800 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 >> 172.21.15.73:0/299955406 conn(0x7feac8101b70 msgr2=0x7feac8194470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 shutdown_connections 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:28.167+0000 7feacd28d700 1 -- 172.21.15.73:0/299955406 wait complete. 2024-03-08T23:56:28.744 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling cephadm module... 2024-03-08T23:56:29.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1052689986' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-08T23:56:29.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setuser ceph since I am not root 2024-03-08T23:56:29.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setgroup ceph since I am not root 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.160+0000 7f01bea3f700 1 Processor -- start 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.160+0000 7f01bea3f700 1 -- start start 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.160+0000 7f01bea3f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.160+0000 7f01bea3f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b8106f20 con 0x7f01b8106530 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36140/0 (socket says 172.21.15.73:36140) 2024-03-08T23:56:30.071 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b7fff700 1 -- 172.21.15.73:0/960623460 learned_addr learned my addr 172.21.15.73:0/960623460 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b7fff700 1 -- 172.21.15.73:0/960623460 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b8107730 con 0x7f01b8106530 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b7fff700 1 --2- 172.21.15.73:0/960623460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0098a0 tx=0x7f01ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=187eaf2968413946 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b6ffd700 1 -- 172.21.15.73:0/960623460 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01ac004160 con 0x7f01b8106530 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b6ffd700 1 -- 172.21.15.73:0/960623460 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01ac004300 con 0x7f01b8106530 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.161+0000 7f01b6ffd700 1 -- 172.21.15.73:0/960623460 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01ac011b50 con 0x7f01b8106530 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 -- 172.21.15.73:0/960623460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 msgr2=0x7f01b8106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 --2- 172.21.15.73:0/960623460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f01ac0098a0 tx=0x7f01ac009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 -- 172.21.15.73:0/960623460 shutdown_connections 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 --2- 172.21.15.73:0/960623460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b8106950 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:30.072 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 -- 172.21.15.73:0/960623460 >> 172.21.15.73:0/960623460 conn(0x7f01b8101b70 msgr2=0x7f01b8103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 -- 172.21.15.73:0/960623460 shutdown_connections 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.162+0000 7f01bea3f700 1 -- 172.21.15.73:0/960623460 wait complete. 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01bea3f700 1 Processor -- start 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01bea3f700 1 -- start start 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01bea3f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01bea3f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01b810aae0 con 0x7f01b8106530 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01b7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01b7fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36154/0 (socket says 172.21.15.73:36154) 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01b7fff700 1 -- 172.21.15.73:0/1052689986 learned_addr learned my addr 172.21.15.73:0/1052689986 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.163+0000 7f01b7fff700 1 -- 172.21.15.73:0/1052689986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01b819aab0 con 0x7f01b8106530 2024-03-08T23:56:30.073 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01b7fff700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009fb0 tx=0x7f01ac011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01ac0038b0 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01ac011e50 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f01ac019550 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01b819ad40 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.164+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01b819da30 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f01ac019800 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01b8063410 con 0x7f01b8106530 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01b57fa700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 0x7f01a003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:30.074 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f01ac04d080 con 0x7f01b8106530 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01b77fe700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 0x7f01a003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.165+0000 7f01b77fe700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 0x7f01a003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006fd0 tx=0x7f01a8006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.174+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f01ac04caa0 con 0x7f01b8106530 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.436+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f01b819dc00 con 0x7f01b8106530 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.487+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f01b819dc00 con 0x7f01b8106530 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.488+0000 7f01b57fa700 1 -- 172.21.15.73:0/1052689986 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f01ac02d960 con 0x7f01b8106530 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 msgr2=0x7f01a003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 0x7f01a003b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f01a8006fd0 tx=0x7f01a8006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 msgr2=0x7f01b819a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f01ac009fb0 tx=0x7f01ac011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 shutdown_connections 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f01a0038ed0 0x7f01a003b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:30.075 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 --2- 172.21.15.73:0/1052689986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f01b8106530 0x7f01b819a570 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:30.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.490+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 >> 172.21.15.73:0/1052689986 conn(0x7f01b8101b70 msgr2=0x7f01b8102720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:30.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.491+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 shutdown_connections 2024-03-08T23:56:30.076 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:29.491+0000 7f01bea3f700 1 -- 172.21.15.73:0/1052689986 wait complete. 2024-03-08T23:56:30.142 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:30.048+0000 7fc6df65b200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:56:30.143 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:30.146+0000 7fc6df65b200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:56:30.719 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1052689986' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-08T23:56:30.719 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-mon[32309]: mgrmap e5: x(active, since 6s) 2024-03-08T23:56:31.037 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:30.803+0000 7fc6df65b200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:56:31.037 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:30.897+0000 7fc6df65b200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:56:31.037 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.040+0000 7fc6df65b200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:56:31.350 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.136+0000 7fc6df65b200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:56:31.350 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.219+0000 7fc6df65b200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:56:31.359 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:31.359 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-08T23:56:31.359 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T23:56:31.359 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.517+0000 7f958446e700 1 Processor -- start 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.517+0000 7f958446e700 1 -- start start 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.517+0000 7f958446e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.517+0000 7f958446e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c07ade0 con 0x7f957c108160 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f958220a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f958220a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36188/0 (socket says 172.21.15.73:36188) 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f958220a700 1 -- 172.21.15.73:0/4158412858 learned_addr learned my addr 172.21.15.73:0/4158412858 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f958220a700 1 -- 172.21.15.73:0/4158412858 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c10aa90 con 0x7f957c108160 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f958220a700 1 --2- 172.21.15.73:0/4158412858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f95700098a0 tx=0x7f9570009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc9a591340362ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f9581208700 1 -- 172.21.15.73:0/4158412858 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9570004160 con 0x7f957c108160 2024-03-08T23:56:31.360 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f9581208700 1 -- 172.21.15.73:0/4158412858 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9570004300 con 0x7f957c108160 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.518+0000 7f9581208700 1 -- 172.21.15.73:0/4158412858 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9570011b50 con 0x7f957c108160 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 -- 172.21.15.73:0/4158412858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 msgr2=0x7f957c10a550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 --2- 172.21.15.73:0/4158412858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f95700098a0 tx=0x7f9570009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 -- 172.21.15.73:0/4158412858 shutdown_connections 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 --2- 172.21.15.73:0/4158412858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c10a550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 -- 172.21.15.73:0/4158412858 >> 172.21.15.73:0/4158412858 conn(0x7f957c101b70 msgr2=0x7f957c103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 -- 172.21.15.73:0/4158412858 shutdown_connections 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.519+0000 7f958446e700 1 -- 172.21.15.73:0/4158412858 wait complete. 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958446e700 1 Processor -- start 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958446e700 1 -- start start 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958446e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958446e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f957c10ba00 con 0x7f957c108160 2024-03-08T23:56:31.361 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958220a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958220a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36196/0 (socket says 172.21.15.73:36196) 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958220a700 1 -- 172.21.15.73:0/691049698 learned_addr learned my addr 172.21.15.73:0/691049698 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.520+0000 7f958220a700 1 -- 172.21.15.73:0/691049698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f957c19bb40 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f958220a700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9570009fb0 tx=0x7f9570011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f95700038b0 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9570011e50 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f957c19bdd0 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9570019420 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.521+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f957c19eac0 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9570019720 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f956f7fe700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 0x7f956803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f957c063410 con 0x7f957c108160 2024-03-08T23:56:31.362 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f9581a09700 1 -- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 msgr2=0x7f956803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f957004d030 con 0x7f957c108160 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.522+0000 7f9581a09700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 0x7f956803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.527+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f957000bc70 con 0x7f957c108160 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.723+0000 7f9581a09700 1 -- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 msgr2=0x7f956803b390 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.723+0000 7f9581a09700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 0x7f956803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.784+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f957c19ec90 con 0x7f957c108160 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.785+0000 7f956f7fe700 1 -- 172.21.15.73:0/691049698 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f957c19ec90 con 0x7f957c108160 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 msgr2=0x7f956803b390 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 0x7f956803b390 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 msgr2=0x7f957c19b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9570009fb0 tx=0x7f9570011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.363 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 shutdown_connections 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f9568038ed0 0x7f956803b390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 --2- 172.21.15.73:0/691049698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f957c108160 0x7f957c19b600 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 >> 172.21.15.73:0/691049698 conn(0x7f957c101b70 msgr2=0x7f957c102670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 shutdown_connections 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:30.787+0000 7f958446e700 1 -- 172.21.15.73:0/691049698 wait complete. 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for the mgr to restart... 2024-03-08T23:56:31.364 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr epoch 5... 2024-03-08T23:56:31.602 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/691049698' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-08T23:56:31.603 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.486+0000 7fc6df65b200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:56:31.603 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.597+0000 7fc6df65b200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:56:31.902 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.684+0000 7fc6df65b200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:56:31.902 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.782+0000 7fc6df65b200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:56:32.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:31.958+0000 7fc6df65b200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:56:33.203 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:32.907+0000 7fc6df65b200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:56:33.203 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:33.035+0000 7fc6df65b200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:56:33.454 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:33.368+0000 7fc6df65b200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:56:33.454 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:33.458+0000 7fc6df65b200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:56:33.832 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:33.651+0000 7fc6df65b200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:56:33.832 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:33.836+0000 7fc6df65b200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:56:34.372 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:34 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:34.121+0000 7fc6df65b200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:56:34.373 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:34 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:34.376+0000 7fc6df65b200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:56:34.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:34 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:34.612+0000 7fc6df65b200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:56:35.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:35 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:35.406+0000 7fc6df65b200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:56:36.279 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:35 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:35.969+0000 7fc6df65b200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:56:36.279 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:36.067+0000 7fc6df65b200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:56:36.279 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:36.283+0000 7fc6df65b200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:56:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: Active manager daemon x restarted 2024-03-08T23:56:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: Activating manager daemon x 2024-03-08T23:56:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: osdmap e2: 0 total, 0 up, 0 in 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: mgrmap e6: x(active, starting, since 0.00353787s) 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: Manager daemon x is now available 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:36 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: Found migration_current of "None". Setting to last migration. 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:37.640 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:37 smithi073 ceph-mon[32309]: mgrmap e7: x(active, since 1.00519s) 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.799+0000 7f62dee2b700 1 Processor -- start 2024-03-08T23:56:37.918 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.799+0000 7f62dee2b700 1 -- start start 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dee2b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dee2b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d807ae60 con 0x7f62d81080b0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dcbc7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dcbc7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36204/0 (socket says 172.21.15.73:36204) 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dcbc7700 1 -- 172.21.15.73:0/232052054 learned_addr learned my addr 172.21.15.73:0/232052054 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dcbc7700 1 -- 172.21.15.73:0/232052054 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d810a9e0 con 0x7f62d81080b0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62dcbc7700 1 --2- 172.21.15.73:0/232052054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0098a0 tx=0x7f62cc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93b8e650517a363 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62d77fe700 1 -- 172.21.15.73:0/232052054 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62cc004160 con 0x7f62d81080b0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62d77fe700 1 -- 172.21.15.73:0/232052054 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62cc004300 con 0x7f62d81080b0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.800+0000 7f62d77fe700 1 -- 172.21.15.73:0/232052054 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62cc011b50 con 0x7f62d81080b0 2024-03-08T23:56:37.919 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 -- 172.21.15.73:0/232052054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 msgr2=0x7f62d810a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 --2- 172.21.15.73:0/232052054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7f62cc0098a0 tx=0x7f62cc009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 -- 172.21.15.73:0/232052054 shutdown_connections 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 --2- 172.21.15.73:0/232052054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d810a4a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 -- 172.21.15.73:0/232052054 >> 172.21.15.73:0/232052054 conn(0x7f62d8101aa0 msgr2=0x7f62d8103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 -- 172.21.15.73:0/232052054 shutdown_connections 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.801+0000 7f62dee2b700 1 -- 172.21.15.73:0/232052054 wait complete. 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dee2b700 1 Processor -- start 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dee2b700 1 -- start start 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dee2b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dee2b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62d810ba00 con 0x7f62d81080b0 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dcbc7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:37.920 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dcbc7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36208/0 (socket says 172.21.15.73:36208) 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dcbc7700 1 -- 172.21.15.73:0/490144578 learned_addr learned my addr 172.21.15.73:0/490144578 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dcbc7700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62d819bab0 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.802+0000 7f62dcbc7700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc011b60 tx=0x7f62cc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.803+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62cc003b20 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.803+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62d819bd40 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.803+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f62cc003cc0 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.803+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f62cc024840 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.803+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62d819ea30 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f62cc022020 con 0x7f62d81080b0 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d5ffb700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:37.921 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62c803bb40 con 0x7f62c8038f70 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f62cc011e00 con 0x7f62d81080b0 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d7fff700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:31.804+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:32.005+0000 7f62d7fff700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:32.005+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:32.406+0000 7f62d7fff700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:32.406+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:33.207+0000 7f62d7fff700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:33.207+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:34.808+0000 7f62d7fff700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/1574449933 2024-03-08T23:56:37.922 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:34.808+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:36.298+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7f62cc024a40 con 0x7f62d81080b0 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:36.298+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 msgr2=0x7f62c803b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:36.298+0000 7f62d5ffb700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.298+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7f62cc04eb00 con 0x7f62d81080b0 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.299+0000 7f62d5ffb700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 0x7f62c803ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.299+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f62cc00fec0 con 0x7f62c803c860 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.299+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 0x7f62c803ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.300+0000 7f62d7fff700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 0x7f62c803ec50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f62c4003a10 tx=0x7f62c40092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.300+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f62cc00fec0 con 0x7f62c803c860 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f62d8063410 con 0x7f62c803c860 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62d5ffb700 1 -- 172.21.15.73:0/490144578 <== mgr.14120 v2:172.21.15.73:6800/4041937587 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f62d8063410 con 0x7f62c803c860 2024-03-08T23:56:37.923 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 msgr2=0x7f62c803ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 0x7f62c803ec50 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f62c4003a10 tx=0x7f62c40092b0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 msgr2=0x7f62d819b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7f62cc011b60 tx=0x7f62cc011c40 comp rx=0 tx=0).stop 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 shutdown_connections 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f62c803c860 0x7f62c803ec50 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f62d81080b0 0x7f62d819b570 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 --2- 172.21.15.73:0/490144578 >> [v2:172.21.15.73:6800/1574449933,v1:172.21.15.73:6801/1574449933] conn(0x7f62c8038f70 0x7f62c803b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 >> 172.21.15.73:0/490144578 conn(0x7f62d8101aa0 msgr2=0x7f62d81025e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 shutdown_connections 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:37.307+0000 7f62dee2b700 1 -- 172.21.15.73:0/490144578 wait complete. 2024-03-08T23:56:37.924 INFO:teuthology.orchestra.run.smithi073.stdout:mgr epoch 5 is available 2024-03-08T23:56:37.925 INFO:teuthology.orchestra.run.smithi073.stdout:Setting orchestrator backend to cephadm... 2024-03-08T23:56:38.554 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:38 smithi073 ceph-mon[32309]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-08T23:56:38.554 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:38 smithi073 ceph-mon[32309]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-08T23:56:38.555 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:38 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:39.173 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52b59e700 1 Processor -- start 2024-03-08T23:56:39.173 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52b59e700 1 -- start start 2024-03-08T23:56:39.173 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52b59e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:39.173 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52b59e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c07ae80 con 0x7fb52c107ef0 2024-03-08T23:56:39.173 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36278/0 (socket says 172.21.15.73:36278) 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52a59c700 1 -- 172.21.15.73:0/3606041556 learned_addr learned my addr 172.21.15.73:0/3606041556 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.339+0000 7fb52a59c700 1 -- 172.21.15.73:0/3606041556 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c10a820 con 0x7fb52c107ef0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52a59c700 1 --2- 172.21.15.73:0/3606041556 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5200098a0 tx=0x7fb520009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d5f128193f3e0fd0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52959a700 1 -- 172.21.15.73:0/3606041556 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb520004160 con 0x7fb52c107ef0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52959a700 1 -- 172.21.15.73:0/3606041556 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb520004300 con 0x7fb52c107ef0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52959a700 1 -- 172.21.15.73:0/3606041556 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb520011b50 con 0x7fb52c107ef0 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 -- 172.21.15.73:0/3606041556 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 msgr2=0x7fb52c10a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 --2- 172.21.15.73:0/3606041556 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fb5200098a0 tx=0x7fb520009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:39.174 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 -- 172.21.15.73:0/3606041556 shutdown_connections 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 --2- 172.21.15.73:0/3606041556 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c10a2e0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 -- 172.21.15.73:0/3606041556 >> 172.21.15.73:0/3606041556 conn(0x7fb52c101940 msgr2=0x7fb52c103d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 -- 172.21.15.73:0/3606041556 shutdown_connections 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.340+0000 7fb52b59e700 1 -- 172.21.15.73:0/3606041556 wait complete. 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52b59e700 1 Processor -- start 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52b59e700 1 -- start start 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52b59e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52b59e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c10b7d0 con 0x7fb52c107ef0 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52a59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36290/0 (socket says 172.21.15.73:36290) 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.341+0000 7fb52a59c700 1 -- 172.21.15.73:0/1032556740 learned_addr learned my addr 172.21.15.73:0/1032556740 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb52a59c700 1 -- 172.21.15.73:0/1032556740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c19b8b0 con 0x7fb52c107ef0 2024-03-08T23:56:39.175 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb52a59c700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb520011b60 tx=0x7fb520011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb520003b20 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c19bba0 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb520003cc0 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb520024840 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.342+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c19e830 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.343+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fb520022020 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.343+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c063410 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.343+0000 7fb51b7fe700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 0x7fb51403b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.343+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb520011e00 con 0x7fb52c107ef0 2024-03-08T23:56:39.176 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.343+0000 7fb529d9b700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 0x7fb51403b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.344+0000 7fb529d9b700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 0x7fb51403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.348+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb520019c70 con 0x7fb52c107ef0 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.353+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb52002d910 con 0x7fb52c107ef0 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.558+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fb52c19ea00 con 0x7fb514038e00 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.563+0000 7fb51b7fe700 1 -- 172.21.15.73:0/1032556740 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb52c19ea00 con 0x7fb514038e00 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 msgr2=0x7fb51403b2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 0x7fb51403b2c0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fb51c006fd0 tx=0x7fb51c006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 msgr2=0x7fb52c19b370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fb520011b60 tx=0x7fb520011c40 comp rx=0 tx=0).stop 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 shutdown_connections 2024-03-08T23:56:39.177 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb514038e00 0x7fb51403b2c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:39.178 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 --2- 172.21.15.73:0/1032556740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb52c107ef0 0x7fb52c19b370 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:39.178 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 >> 172.21.15.73:0/1032556740 conn(0x7fb52c101940 msgr2=0x7fb52c1023e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:39.178 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.565+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 shutdown_connections 2024-03-08T23:56:39.178 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:38.566+0000 7fb52b59e700 1 -- 172.21.15.73:0/1032556740 wait complete. 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: [08/Mar/2024:23:56:37] ENGINE Bus STARTING 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: [08/Mar/2024:23:56:37] ENGINE Serving on https://172.21.15.73:7150 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: [08/Mar/2024:23:56:37] ENGINE Serving on http://172.21.15.73:8765 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: [08/Mar/2024:23:56:37] ENGINE Bus STARTED 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: mgrmap e8: x(active, since 2s) 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:39.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:39 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:40.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:40 smithi073 ceph-mon[32309]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.636+0000 7f415d764700 1 Processor -- start 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.636+0000 7f415d764700 1 -- start start 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.636+0000 7f415d764700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.636+0000 7f415d764700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4158106ed0 con 0x7f41581064e0 2024-03-08T23:56:40.451 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:40.452 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33502/0 (socket says 172.21.15.73:33502) 2024-03-08T23:56:40.452 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f4156ffd700 1 -- 172.21.15.73:0/583180547 learned_addr learned my addr 172.21.15.73:0/583180547 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:40.452 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f4156ffd700 1 -- 172.21.15.73:0/583180547 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41581076e0 con 0x7f41581064e0 2024-03-08T23:56:40.452 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f4156ffd700 1 --2- 172.21.15.73:0/583180547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41480098a0 tx=0x7f4148009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bef6220d26b07e9 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:40.459 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f41567fc700 1 -- 172.21.15.73:0/583180547 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4148004160 con 0x7f41581064e0 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f41567fc700 1 -- 172.21.15.73:0/583180547 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4148004300 con 0x7f41581064e0 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.637+0000 7f41567fc700 1 -- 172.21.15.73:0/583180547 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4148011b50 con 0x7f41581064e0 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 -- 172.21.15.73:0/583180547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 msgr2=0x7f4158106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 --2- 172.21.15.73:0/583180547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f41480098a0 tx=0x7f4148009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 -- 172.21.15.73:0/583180547 shutdown_connections 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 --2- 172.21.15.73:0/583180547 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f4158106900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 -- 172.21.15.73:0/583180547 >> 172.21.15.73:0/583180547 conn(0x7f4158101aa0 msgr2=0x7f4158103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 -- 172.21.15.73:0/583180547 shutdown_connections 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 -- 172.21.15.73:0/583180547 wait complete. 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.638+0000 7f415d764700 1 Processor -- start 2024-03-08T23:56:40.460 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f415d764700 1 -- start start 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f415d764700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f415d764700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f415810aa90 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4156ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33504/0 (socket says 172.21.15.73:33504) 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4156ffd700 1 -- 172.21.15.73:0/1818907392 learned_addr learned my addr 172.21.15.73:0/1818907392 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4156ffd700 1 -- 172.21.15.73:0/1818907392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f415819e370 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4156ffd700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4148011670 tx=0x7f4148011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41480039c0 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4148011e50 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f415819e540 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f414801a5c0 con 0x7f41581064e0 2024-03-08T23:56:40.461 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.639+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f415819e8e0 con 0x7f41581064e0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.640+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f414801a8c0 con 0x7f41581064e0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.640+0000 7f4154ff9700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 0x7f414403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.640+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4158063410 con 0x7f41581064e0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.641+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f414804d520 con 0x7f41581064e0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.641+0000 7f414e7fc700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 0x7f414403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.641+0000 7f414e7fc700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 0x7f414403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4140006fd0 tx=0x7f4140006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.645+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4148019c70 con 0x7f41581064e0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.844+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f4158102d20 con 0x7f4144038ed0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.845+0000 7f4154ff9700 1 -- 172.21.15.73:0/1818907392 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f4158102d20 con 0x7f4144038ed0 2024-03-08T23:56:40.462 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 msgr2=0x7f414403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 0x7f414403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f4140006fd0 tx=0x7f4140006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 msgr2=0x7f415807aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f4148011670 tx=0x7f4148011b60 comp rx=0 tx=0).stop 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 shutdown_connections 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f4144038ed0 0x7f414403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 --2- 172.21.15.73:0/1818907392 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41581064e0 0x7f415807aa40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 >> 172.21.15.73:0/1818907392 conn(0x7f4158101aa0 msgr2=0x7f4158102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.847+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 shutdown_connections 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:39.848+0000 7f415d764700 1 -- 172.21.15.73:0/1818907392 wait complete. 2024-03-08T23:56:40.463 INFO:teuthology.orchestra.run.smithi073.stdout:Generating ssh key... 2024-03-08T23:56:41.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-mon[32309]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: Generating public/private rsa key pair. 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: Your identification has been saved in /tmp/tmppxqvb17b/key. 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: Your public key has been saved in /tmp/tmppxqvb17b/key.pub. 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: The key fingerprint is: 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: SHA256:S3ftkwtkYn9g/RNffaewPjW+szUZuR36pnz8pxa+JbA ceph-44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: The key's randomart image is: 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: +---[RSA 3072]----+ 2024-03-08T23:56:41.653 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | | 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | | 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | | 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | o o| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | S + O o==| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | . + B B=+%| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | . E=OB*| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | .oo*X+| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: | .=XB+| 2024-03-08T23:56:41.654 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: +----[SHA256]-----+ 2024-03-08T23:56:42.112 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26ba93700 1 Processor -- start 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26ba93700 1 -- start start 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26ba93700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26ba93700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe264106ed0 con 0x7fe2641064e0 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26982f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26982f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33510/0 (socket says 172.21.15.73:33510) 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26982f700 1 -- 172.21.15.73:0/3093206988 learned_addr learned my addr 172.21.15.73:0/3093206988 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.894+0000 7fe26982f700 1 -- 172.21.15.73:0/3093206988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2641076e0 con 0x7fe2641064e0 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26982f700 1 --2- 172.21.15.73:0/3093206988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2600098a0 tx=0x7fe260009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=553e9246a8f1b9ea server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26882d700 1 -- 172.21.15.73:0/3093206988 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe260004160 con 0x7fe2641064e0 2024-03-08T23:56:42.113 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26882d700 1 -- 172.21.15.73:0/3093206988 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe260004300 con 0x7fe2641064e0 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26882d700 1 -- 172.21.15.73:0/3093206988 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe260011b50 con 0x7fe2641064e0 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26ba93700 1 -- 172.21.15.73:0/3093206988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 msgr2=0x7fe264106900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26ba93700 1 --2- 172.21.15.73:0/3093206988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7fe2600098a0 tx=0x7fe260009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26ba93700 1 -- 172.21.15.73:0/3093206988 shutdown_connections 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26ba93700 1 --2- 172.21.15.73:0/3093206988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe264106900 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.895+0000 7fe26ba93700 1 -- 172.21.15.73:0/3093206988 >> 172.21.15.73:0/3093206988 conn(0x7fe264101aa0 msgr2=0x7fe264103ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 -- 172.21.15.73:0/3093206988 shutdown_connections 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 -- 172.21.15.73:0/3093206988 wait complete. 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 Processor -- start 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 -- start start 2024-03-08T23:56:42.114 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26ba93700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe26410aa90 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26982f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26982f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33526/0 (socket says 172.21.15.73:33526) 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.896+0000 7fe26982f700 1 -- 172.21.15.73:0/1109931891 learned_addr learned my addr 172.21.15.73:0/1109931891 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe26982f700 1 -- 172.21.15.73:0/1109931891 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe26419aa60 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe26982f700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe260011670 tx=0x7fe260011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2600039c0 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe260011e50 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe26419acf0 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe26001a5c0 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.897+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe26419d9e0 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.898+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe264063410 con 0x7fe2641064e0 2024-03-08T23:56:42.115 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.898+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe260024430 con 0x7fe2641064e0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.898+0000 7fe25affd700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 0x7fe25003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.898+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe26004d320 con 0x7fe2641064e0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.898+0000 7fe26902e700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 0x7fe25003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.899+0000 7fe26902e700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 0x7fe25003b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:40.903+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe260017ca0 con 0x7fe2641064e0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.098+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fe264102d20 con 0x7fe250038ed0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.558+0000 7fe25affd700 1 -- 172.21.15.73:0/1109931891 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fe264102d20 con 0x7fe250038ed0 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 msgr2=0x7fe25003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 0x7fe25003b390 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fe254006fd0 tx=0x7fe254006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 msgr2=0x7fe26419a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fe260011670 tx=0x7fe260011b60 comp rx=0 tx=0).stop 2024-03-08T23:56:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 shutdown_connections 2024-03-08T23:56:42.117 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe250038ed0 0x7fe25003b390 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:42.117 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 --2- 172.21.15.73:0/1109931891 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe2641064e0 0x7fe26419a520 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:42.117 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 >> 172.21.15.73:0/1109931891 conn(0x7fe264101aa0 msgr2=0x7fe264102610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:42.117 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 shutdown_connections 2024-03-08T23:56:42.117 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:41.560+0000 7fe26ba93700 1 -- 172.21.15.73:0/1109931891 wait complete. 2024-03-08T23:56:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:42 smithi073 ceph-mon[32309]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:42 smithi073 ceph-mon[32309]: Generating ssh key... 2024-03-08T23:56:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:42 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:42.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:42 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.564+0000 7f2a2ce12700 1 Processor -- start 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.564+0000 7f2a2ce12700 1 -- start start 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.564+0000 7f2a2ce12700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.564+0000 7f2a2ce12700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a28106f40 con 0x7f2a28106550 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33540/0 (socket says 172.21.15.73:33540) 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2659c700 1 -- 172.21.15.73:0/3846962650 learned_addr learned my addr 172.21.15.73:0/3846962650 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2659c700 1 -- 172.21.15.73:0/3846962650 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28107750 con 0x7f2a28106550 2024-03-08T23:56:43.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2659c700 1 --2- 172.21.15.73:0/3846962650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a180098a0 tx=0x7f2a18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525b798c9da6283 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2559a700 1 -- 172.21.15.73:0/3846962650 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18004160 con 0x7f2a28106550 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2559a700 1 -- 172.21.15.73:0/3846962650 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a18004300 con 0x7f2a28106550 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.565+0000 7f2a2559a700 1 -- 172.21.15.73:0/3846962650 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18011b50 con 0x7f2a28106550 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 -- 172.21.15.73:0/3846962650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 msgr2=0x7f2a28106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/3846962650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f2a180098a0 tx=0x7f2a18009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 -- 172.21.15.73:0/3846962650 shutdown_connections 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/3846962650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a28106970 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 -- 172.21.15.73:0/3846962650 >> 172.21.15.73:0/3846962650 conn(0x7f2a28101b70 msgr2=0x7f2a28103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 -- 172.21.15.73:0/3846962650 shutdown_connections 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.566+0000 7f2a2ce12700 1 -- 172.21.15.73:0/3846962650 wait complete. 2024-03-08T23:56:43.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2ce12700 1 Processor -- start 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2ce12700 1 -- start start 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2ce12700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2ce12700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a2810ab00 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2659c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33548/0 (socket says 172.21.15.73:33548) 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2659c700 1 -- 172.21.15.73:0/2344787813 learned_addr learned my addr 172.21.15.73:0/2344787813 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2659c700 1 -- 172.21.15.73:0/2344787813 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a2819aa80 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.567+0000 7f2a2659c700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009fb0 tx=0x7f2a18011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a180038b0 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a2819ad10 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a2819da00 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a18011e50 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2a18019550 con 0x7f2a28106550 2024-03-08T23:56:43.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.568+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f2a180196f0 con 0x7f2a28106550 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.569+0000 7f2a177fe700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 0x7f2a1003d5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.569+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a08005360 con 0x7f2a28106550 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.569+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f2a1804d190 con 0x7f2a28106550 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.569+0000 7f2a25d9b700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 0x7f2a1003d5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.569+0000 7f2a25d9b700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 0x7f2a1003d5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006fd0 tx=0x7f2a1c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.574+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2a1800bc70 con 0x7f2a28106550 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.770+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a08000bf0 con 0x7f2a1003b0e0 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.771+0000 7f2a177fe700 1 -- 172.21.15.73:0/2344787813 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f2a08000bf0 con 0x7f2a1003b0e0 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 msgr2=0x7f2a1003d5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 0x7f2a1003d5a0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f2a1c006fd0 tx=0x7f2a1c006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 msgr2=0x7f2a2819a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f2a18009fb0 tx=0x7f2a18011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 shutdown_connections 2024-03-08T23:56:43.405 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f2a1003b0e0 0x7f2a1003d5a0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 --2- 172.21.15.73:0/2344787813 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2a28106550 0x7f2a2819a540 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 >> 172.21.15.73:0/2344787813 conn(0x7f2a28101b70 msgr2=0x7f2a281026e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 shutdown_connections 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:42.773+0000 7f2a2ce12700 1 -- 172.21.15.73:0/2344787813 wait complete. 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:Adding key to root@localhost authorized_keys... 2024-03-08T23:56:43.406 INFO:teuthology.orchestra.run.smithi073.stdout:Adding host smithi073... 2024-03-08T23:56:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:43 smithi073 ceph-mon[32309]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:44.860 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:44 smithi073 ceph-mon[32309]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi073", "addr": "172.21.15.73", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:45 smithi073 ceph-mon[32309]: Deploying cephadm binary to smithi073 2024-03-08T23:56:48.827 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Added host 'smithi073' with addr '172.21.15.73' 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe824fae700 1 Processor -- start 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe824fae700 1 -- start start 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe824fae700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe824fae700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe820106f40 con 0x7fe820106550 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33552/0 (socket says 172.21.15.73:33552) 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.789+0000 7fe81e59c700 1 -- 172.21.15.73:0/520742943 learned_addr learned my addr 172.21.15.73:0/520742943 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe81e59c700 1 -- 172.21.15.73:0/520742943 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe820107750 con 0x7fe820106550 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe81e59c700 1 --2- 172.21.15.73:0/520742943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe8100098a0 tx=0x7fe810009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18da80757ee59d9c server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe81d59a700 1 -- 172.21.15.73:0/520742943 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe810004160 con 0x7fe820106550 2024-03-08T23:56:48.828 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe81d59a700 1 -- 172.21.15.73:0/520742943 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe810004300 con 0x7fe820106550 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe81d59a700 1 -- 172.21.15.73:0/520742943 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe810011b50 con 0x7fe820106550 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe824fae700 1 -- 172.21.15.73:0/520742943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 msgr2=0x7fe820106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.790+0000 7fe824fae700 1 --2- 172.21.15.73:0/520742943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe8100098a0 tx=0x7fe810009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 -- 172.21.15.73:0/520742943 shutdown_connections 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 --2- 172.21.15.73:0/520742943 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe820106970 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 -- 172.21.15.73:0/520742943 >> 172.21.15.73:0/520742943 conn(0x7fe820101b70 msgr2=0x7fe820103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 -- 172.21.15.73:0/520742943 shutdown_connections 2024-03-08T23:56:48.829 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 -- 172.21.15.73:0/520742943 wait complete. 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 Processor -- start 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.791+0000 7fe824fae700 1 -- start start 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe824fae700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe824fae700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82010ab00 con 0x7fe820106550 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe81e59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33560/0 (socket says 172.21.15.73:33560) 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe81e59c700 1 -- 172.21.15.73:0/3158303134 learned_addr learned my addr 172.21.15.73:0/3158303134 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe81e59c700 1 -- 172.21.15.73:0/3158303134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82019aa80 con 0x7fe820106550 2024-03-08T23:56:48.830 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe81e59c700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe810009fb0 tx=0x7fe810011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe8100038b0 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82019ad10 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe810011e50 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe810019550 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.792+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82019da00 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.793+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fe810019850 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.793+0000 7fe80f7fe700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 0x7fe80803b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.793+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fe81004d2a0 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.793+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe820063410 con 0x7fe820106550 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.793+0000 7fe81dd9b700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 0x7fe80803b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:48.831 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.794+0000 7fe81dd9b700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 0x7fe80803b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe814006fd0 tx=0x7fe814006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.798+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe810003ac0 con 0x7fe820106550 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:43.997+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi073", "addr": "172.21.15.73", "target": ["mon-mgr", ""]}) v1 -- 0x7fe820102db0 con 0x7fe808038ed0 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.220+0000 7fe80f7fe700 1 -- 172.21.15.73:0/3158303134 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fe820102db0 con 0x7fe808038ed0 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 msgr2=0x7fe80803b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 0x7fe80803b390 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fe814006fd0 tx=0x7fe814006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 msgr2=0x7fe82019a540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe810009fb0 tx=0x7fe810011bf0 comp rx=0 tx=0).stop 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 shutdown_connections 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fe808038ed0 0x7fe80803b390 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 --2- 172.21.15.73:0/3158303134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe820106550 0x7fe82019a540 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 >> 172.21.15.73:0/3158303134 conn(0x7fe820101b70 msgr2=0x7fe8201026a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 shutdown_connections 2024-03-08T23:56:48.832 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:48.223+0000 7fe824fae700 1 -- 172.21.15.73:0/3158303134 wait complete. 2024-03-08T23:56:48.833 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying unmanaged mon service... 2024-03-08T23:56:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:49 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:49 smithi073 ceph-mon[32309]: Added host smithi073 2024-03-08T23:56:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:49 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.251+0000 7f14bd430700 1 Processor -- start 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.251+0000 7f14bd430700 1 -- start start 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14bd430700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14bd430700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b8108050 con 0x7f14b8107660 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56396/0 (socket says 172.21.15.73:56396) 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b6ffd700 1 -- 172.21.15.73:0/4248561923 learned_addr learned my addr 172.21.15.73:0/4248561923 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b6ffd700 1 -- 172.21.15.73:0/4248561923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b8108810 con 0x7f14b8107660 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b6ffd700 1 --2- 172.21.15.73:0/4248561923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0098a0 tx=0x7f14ac009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2663959f8e5a8948 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:50.196 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b5ffb700 1 -- 172.21.15.73:0/4248561923 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac004160 con 0x7f14b8107660 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b5ffb700 1 -- 172.21.15.73:0/4248561923 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14ac004300 con 0x7f14b8107660 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.252+0000 7f14b5ffb700 1 -- 172.21.15.73:0/4248561923 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac011b50 con 0x7f14b8107660 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 -- 172.21.15.73:0/4248561923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 msgr2=0x7f14b8107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 --2- 172.21.15.73:0/4248561923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f14ac0098a0 tx=0x7f14ac009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 -- 172.21.15.73:0/4248561923 shutdown_connections 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 --2- 172.21.15.73:0/4248561923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b8107a80 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 -- 172.21.15.73:0/4248561923 >> 172.21.15.73:0/4248561923 conn(0x7f14b8102c40 msgr2=0x7f14b8105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 -- 172.21.15.73:0/4248561923 shutdown_connections 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.253+0000 7f14bd430700 1 -- 172.21.15.73:0/4248561923 wait complete. 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14bd430700 1 Processor -- start 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14bd430700 1 -- start start 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14bd430700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14bd430700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14b810bbc0 con 0x7f14b8107660 2024-03-08T23:56:50.197 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14b6ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56404/0 (socket says 172.21.15.73:56404) 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14b6ffd700 1 -- 172.21.15.73:0/2506043437 learned_addr learned my addr 172.21.15.73:0/2506043437 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14b6ffd700 1 -- 172.21.15.73:0/2506043437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b819bb40 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f14b6ffd700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f14ac011850 tx=0x7f14ac011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.254+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac003af0 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f14ac020070 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f14ac0195a0 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14b819bdd0 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14b819eac0 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f14ac0198a0 con 0x7f14b8107660 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.255+0000 7f149ffff700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 0x7f14a003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:50.198 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.256+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f14ac011e00 con 0x7f14b8107660 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.256+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14b8063410 con 0x7f14b8107660 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.256+0000 7f14b67fc700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 0x7f14a003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.256+0000 7f14b67fc700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 0x7f14a003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006fd0 tx=0x7f14a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.261+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f14ac003c90 con 0x7f14b8107660 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.496+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f14b8103eb0 con 0x7f14a0038f20 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.504+0000 7f149ffff700 1 -- 172.21.15.73:0/2506043437 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f14b8103eb0 con 0x7f14a0038f20 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 msgr2=0x7f14a003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 0x7f14a003b3e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f14a8006fd0 tx=0x7f14a8006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 msgr2=0x7f14b819b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7f14ac011850 tx=0x7f14ac011c10 comp rx=0 tx=0).stop 2024-03-08T23:56:50.199 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 shutdown_connections 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f14a0038f20 0x7f14a003b3e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 --2- 172.21.15.73:0/2506043437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14b8107660 0x7f14b819b600 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.506+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 >> 172.21.15.73:0/2506043437 conn(0x7f14b8102c40 msgr2=0x7f14b81037a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.507+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 shutdown_connections 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:49.507+0000 7f14bd430700 1 -- 172.21.15.73:0/2506043437 wait complete. 2024-03-08T23:56:50.200 INFO:teuthology.orchestra.run.smithi073.stdout:Deploying unmanaged mgr service... 2024-03-08T23:56:50.782 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:50 smithi073 ceph-mon[32309]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:50.782 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:50 smithi073 ceph-mon[32309]: Saving service mon spec with placement count:5 2024-03-08T23:56:50.782 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:50 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:51 smithi073 ceph-mon[32309]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:56:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:51 smithi073 ceph-mon[32309]: Saving service mgr spec with placement count:2 2024-03-08T23:56:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:51 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:52.391 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-08T23:56:52.391 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.689+0000 7fccbb172700 1 Processor -- start 2024-03-08T23:56:52.391 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.689+0000 7fccbb172700 1 -- start start 2024-03-08T23:56:52.391 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.689+0000 7fccbb172700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.689+0000 7fccbb172700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb41062d0 con 0x7fccb41058e0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.689+0000 7fccb8f0e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb8f0e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56410/0 (socket says 172.21.15.73:56410) 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb8f0e700 1 -- 172.21.15.73:0/1386559654 learned_addr learned my addr 172.21.15.73:0/1386559654 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb8f0e700 1 -- 172.21.15.73:0/1386559654 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4108870 con 0x7fccb41058e0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb8f0e700 1 --2- 172.21.15.73:0/1386559654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcca80098a0 tx=0x7fcca8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2df406d0a8075c34 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb37fe700 1 -- 172.21.15.73:0/1386559654 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca8004160 con 0x7fccb41058e0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb37fe700 1 -- 172.21.15.73:0/1386559654 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcca8004300 con 0x7fccb41058e0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.690+0000 7fccb37fe700 1 -- 172.21.15.73:0/1386559654 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca8011b50 con 0x7fccb41058e0 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 -- 172.21.15.73:0/1386559654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 msgr2=0x7fccb4105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 --2- 172.21.15.73:0/1386559654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcca80098a0 tx=0x7fcca8009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 -- 172.21.15.73:0/1386559654 shutdown_connections 2024-03-08T23:56:52.392 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 --2- 172.21.15.73:0/1386559654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb4105d00 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 -- 172.21.15.73:0/1386559654 >> 172.21.15.73:0/1386559654 conn(0x7fccb41011d0 msgr2=0x7fccb4103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 -- 172.21.15.73:0/1386559654 shutdown_connections 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 -- 172.21.15.73:0/1386559654 wait complete. 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.691+0000 7fccbb172700 1 Processor -- start 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccbb172700 1 -- start start 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccbb172700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccbb172700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb410bc20 con 0x7fccb41058e0 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb8f0e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb8f0e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56424/0 (socket says 172.21.15.73:56424) 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb8f0e700 1 -- 172.21.15.73:0/4144872970 learned_addr learned my addr 172.21.15.73:0/4144872970 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb8f0e700 1 -- 172.21.15.73:0/4144872970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb419bb40 con 0x7fccb41058e0 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb8f0e700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcca8000c00 tx=0x7fcca8011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:52.393 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca8003b20 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcca8003cc0 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fcca8024910 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.692+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb419bdd0 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.693+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb419eac0 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fcca8022020 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccb1ffb700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 0x7fcca403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fcca8011e00 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccb3fff700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 0x7fcca403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4063410 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.694+0000 7fccb3fff700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 0x7fcca403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcca0006fd0 tx=0x7fcca0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.702+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcca802cdf0 con 0x7fccb41058e0 2024-03-08T23:56:52.394 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.916+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fccb41024d0 con 0x7fcca4038f70 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.923+0000 7fccb1ffb700 1 -- 172.21.15.73:0/4144872970 <== mgr.14120 v2:172.21.15.73:6800/4041937587 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fccb41024d0 con 0x7fcca4038f70 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 msgr2=0x7fcca403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 0x7fcca403b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcca0006fd0 tx=0x7fcca0006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 msgr2=0x7fccb419b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcca8000c00 tx=0x7fcca8011c10 comp rx=0 tx=0).stop 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 shutdown_connections 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fcca4038f70 0x7fcca403b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 --2- 172.21.15.73:0/4144872970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fccb41058e0 0x7fccb419b600 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 >> 172.21.15.73:0/4144872970 conn(0x7fccb41011d0 msgr2=0x7fccb4101dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:52.395 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 shutdown_connections 2024-03-08T23:56:52.396 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:50.927+0000 7fccbb172700 1 -- 172.21.15.73:0/4144872970 wait complete. 2024-03-08T23:56:53.289 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:52 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:53.289 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:52 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c99f9e700 1 Processor -- start 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c99f9e700 1 -- start start 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c99f9e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c99f9e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c94108490 con 0x7f9c94107aa0 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c937fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c937fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56426/0 (socket says 172.21.15.73:56426) 2024-03-08T23:56:54.310 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c937fe700 1 -- 172.21.15.73:0/2951125023 learned_addr learned my addr 172.21.15.73:0/2951125023 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c937fe700 1 -- 172.21.15.73:0/2951125023 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94106700 con 0x7f9c94107aa0 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.820+0000 7f9c937fe700 1 --2- 172.21.15.73:0/2951125023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c84009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e15363704e00c7a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c927fc700 1 -- 172.21.15.73:0/2951125023 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c84004160 con 0x7f9c94107aa0 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c927fc700 1 -- 172.21.15.73:0/2951125023 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c84004300 con 0x7f9c94107aa0 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c927fc700 1 -- 172.21.15.73:0/2951125023 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c84011b20 con 0x7f9c94107aa0 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 -- 172.21.15.73:0/2951125023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 msgr2=0x7f9c94107ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/2951125023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f9c840098a0 tx=0x7f9c84009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 -- 172.21.15.73:0/2951125023 shutdown_connections 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/2951125023 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c94107ec0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 -- 172.21.15.73:0/2951125023 >> 172.21.15.73:0/2951125023 conn(0x7f9c941011d0 msgr2=0x7f9c94103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 -- 172.21.15.73:0/2951125023 shutdown_connections 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.821+0000 7f9c99f9e700 1 -- 172.21.15.73:0/2951125023 wait complete. 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c99f9e700 1 Processor -- start 2024-03-08T23:56:54.311 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c99f9e700 1 -- start start 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c99f9e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c99f9e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c9410aac0 con 0x7f9c94107aa0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c937fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c937fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56430/0 (socket says 172.21.15.73:56430) 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.822+0000 7f9c937fe700 1 -- 172.21.15.73:0/613507357 learned_addr learned my addr 172.21.15.73:0/613507357 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c937fe700 1 -- 172.21.15.73:0/613507357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c94191c00 con 0x7f9c94107aa0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c937fe700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c840117e0 tx=0x7f9c84011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c84003ba0 con 0x7f9c94107aa0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c8401a910 con 0x7f9c94107aa0 2024-03-08T23:56:54.312 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c9418fdf0 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9c840248f0 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.823+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c941902b0 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.824+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9c84022020 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.824+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c94063410 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.824+0000 7f9c90ff9700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 0x7f9c7c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.824+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9c8404d840 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.824+0000 7f9c92ffd700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 0x7f9c7c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.825+0000 7f9c92ffd700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 0x7f9c7c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9c88006fd0 tx=0x7f9c88006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:52.829+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9c84018350 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.095+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f9c94195400 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.099+0000 7f9c90ff9700 1 -- 172.21.15.73:0/613507357 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7f9c84019be0 con 0x7f9c94107aa0 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 msgr2=0x7f9c7c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 0x7f9c7c03b3e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f9c88006fd0 tx=0x7f9c88006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 msgr2=0x7f9c941916c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:54.313 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f9c840117e0 tx=0x7f9c84011ba0 comp rx=0 tx=0).stop 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 shutdown_connections 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f9c7c038f20 0x7f9c7c03b3e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 --2- 172.21.15.73:0/613507357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c94107aa0 0x7f9c941916c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.101+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 >> 172.21.15.73:0/613507357 conn(0x7f9c941011d0 msgr2=0x7f9c94102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.102+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 shutdown_connections 2024-03-08T23:56:54.314 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:53.102+0000 7f9c99f9e700 1 -- 172.21.15.73:0/613507357 wait complete. 2024-03-08T23:56:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:54 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/613507357' entity='client.admin' 2024-03-08T23:56:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:54 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02f476700 1 Processor -- start 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02f476700 1 -- start start 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02f476700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02f476700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd028108000 con 0x7fd028107610 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02d212700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02d212700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56446/0 (socket says 172.21.15.73:56446) 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02d212700 1 -- 172.21.15.73:0/1584199036 learned_addr learned my addr 172.21.15.73:0/1584199036 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.754+0000 7fd02d212700 1 -- 172.21.15.73:0/1584199036 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0281087c0 con 0x7fd028107610 2024-03-08T23:56:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02d212700 1 --2- 172.21.15.73:0/1584199036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0240098a0 tx=0x7fd024009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3f2cff81c9c225ae server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd01ffff700 1 -- 172.21.15.73:0/1584199036 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd024004160 con 0x7fd028107610 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd01ffff700 1 -- 172.21.15.73:0/1584199036 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd024004300 con 0x7fd028107610 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd01ffff700 1 -- 172.21.15.73:0/1584199036 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd024011b20 con 0x7fd028107610 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02f476700 1 -- 172.21.15.73:0/1584199036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 msgr2=0x7fd028107a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02f476700 1 --2- 172.21.15.73:0/1584199036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd0240098a0 tx=0x7fd024009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02f476700 1 -- 172.21.15.73:0/1584199036 shutdown_connections 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02f476700 1 --2- 172.21.15.73:0/1584199036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd028107a30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.755+0000 7fd02f476700 1 -- 172.21.15.73:0/1584199036 >> 172.21.15.73:0/1584199036 conn(0x7fd028102c50 msgr2=0x7fd028105070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 -- 172.21.15.73:0/1584199036 shutdown_connections 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 -- 172.21.15.73:0/1584199036 wait complete. 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 Processor -- start 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 -- start start 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:55.536 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02f476700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd02810bb70 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02d212700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02d212700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56458/0 (socket says 172.21.15.73:56458) 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.756+0000 7fd02d212700 1 -- 172.21.15.73:0/3091016092 learned_addr learned my addr 172.21.15.73:0/3091016092 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd02d212700 1 -- 172.21.15.73:0/3091016092 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd02819dc90 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd02d212700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0240117e0 tx=0x7fd024011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd024003bf0 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd02401a910 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd02819df20 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fd024024970 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.757+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd02819e380 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fd024022020 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd01e7fc700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 0x7fd01403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd028063410 con 0x7fd028107610 2024-03-08T23:56:55.537 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fd02404d1a0 con 0x7fd028107610 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd02ca11700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 0x7fd01403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.758+0000 7fd02ca11700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 0x7fd01403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd018006fd0 tx=0x7fd018006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.763+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd02400b3e0 con 0x7fd028107610 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.954+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fd0281a0d50 con 0x7fd028107610 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.957+0000 7fd01e7fc700 1 -- 172.21.15.73:0/3091016092 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fd0240172b0 con 0x7fd028107610 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 msgr2=0x7fd01403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 0x7fd01403b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fd018006fd0 tx=0x7fd018006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 msgr2=0x7fd02819d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd0240117e0 tx=0x7fd024011ba0 comp rx=0 tx=0).stop 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 shutdown_connections 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fd014038f70 0x7fd01403b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 --2- 172.21.15.73:0/3091016092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd028107610 0x7fd02819d750 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:55.538 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 >> 172.21.15.73:0/3091016092 conn(0x7fd028102c50 msgr2=0x7fd028103670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:55.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 shutdown_connections 2024-03-08T23:56:55.539 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:54.959+0000 7fd02f476700 1 -- 172.21.15.73:0/3091016092 wait complete. 2024-03-08T23:56:55.539 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling the dashboard module... 2024-03-08T23:56:56.257 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:55 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3091016092' entity='client.admin' 2024-03-08T23:56:56.257 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:55 smithi073 ceph-mon[32309]: from='mgr.14120 172.21.15.73:0/3758283280' entity='mgr.x' 2024-03-08T23:56:57.279 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:56 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/108109882' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-08T23:56:57.279 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:56 smithi073 ceph-mon[32309]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:56:57.279 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setuser ceph since I am not root 2024-03-08T23:56:57.279 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setgroup ceph since I am not root 2024-03-08T23:56:57.619 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:57.530+0000 7fb435ac6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:56:57.620 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:57.623+0000 7fb435ac6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:56:57.664 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.996+0000 7fb2357ca700 1 Processor -- start 2024-03-08T23:56:57.664 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.996+0000 7fb2357ca700 1 -- start start 2024-03-08T23:56:57.664 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.996+0000 7fb2357ca700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:57.664 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.996+0000 7fb2357ca700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2301062d0 con 0x7fb2301058e0 2024-03-08T23:56:57.664 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56468/0 (socket says 172.21.15.73:56468) 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22effd700 1 -- 172.21.15.73:0/1577450417 learned_addr learned my addr 172.21.15.73:0/1577450417 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22effd700 1 -- 172.21.15.73:0/1577450417 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230108870 con 0x7fb2301058e0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22effd700 1 --2- 172.21.15.73:0/1577450417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2180098a0 tx=0x7fb218009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3fd7eab9f1b157a server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22dffb700 1 -- 172.21.15.73:0/1577450417 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb218004160 con 0x7fb2301058e0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22dffb700 1 -- 172.21.15.73:0/1577450417 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb218004300 con 0x7fb2301058e0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.997+0000 7fb22dffb700 1 -- 172.21.15.73:0/1577450417 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb218011b20 con 0x7fb2301058e0 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 -- 172.21.15.73:0/1577450417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 msgr2=0x7fb230105d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 --2- 172.21.15.73:0/1577450417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7fb2180098a0 tx=0x7fb218009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 -- 172.21.15.73:0/1577450417 shutdown_connections 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 --2- 172.21.15.73:0/1577450417 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb230105d00 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 -- 172.21.15.73:0/1577450417 >> 172.21.15.73:0/1577450417 conn(0x7fb2301011d0 msgr2=0x7fb230103650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 -- 172.21.15.73:0/1577450417 shutdown_connections 2024-03-08T23:56:57.665 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.998+0000 7fb2357ca700 1 -- 172.21.15.73:0/1577450417 wait complete. 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb2357ca700 1 Processor -- start 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb2357ca700 1 -- start start 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb2357ca700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb2357ca700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb23010bc20 con 0x7fb2301058e0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb22effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb22effd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56478/0 (socket says 172.21.15.73:56478) 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb22effd700 1 -- 172.21.15.73:0/108109882 learned_addr learned my addr 172.21.15.73:0/108109882 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:55.999+0000 7fb22effd700 1 -- 172.21.15.73:0/108109882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb230108630 con 0x7fb2301058e0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb22effd700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2180117e0 tx=0x7fb218011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb218003bf0 con 0x7fb2301058e0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb21801a910 con 0x7fb2301058e0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb230106820 con 0x7fb2301058e0 2024-03-08T23:56:57.666 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb218024970 con 0x7fb2301058e0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.000+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb230106ce0 con 0x7fb2301058e0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb218022020 con 0x7fb2301058e0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb227fff700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 0x7fb21c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb230063410 con 0x7fb2301058e0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb21804d620 con 0x7fb2301058e0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb22e7fc700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 0x7fb21c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.001+0000 7fb22e7fc700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 0x7fb21c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:57.667 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.006+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb2180172b0 con 0x7fb2301058e0 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.261+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb230107630 con 0x7fb2301058e0 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.960+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb21802cdf0 con 0x7fb2301058e0 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.962+0000 7fb227fff700 1 -- 172.21.15.73:0/108109882 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb21802dc90 con 0x7fb2301058e0 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.963+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 msgr2=0x7fb21c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.963+0000 7fb2357ca700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 0x7fb21c03b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7fb220006fd0 tx=0x7fb220006e40 comp rx=0 tx=0).stop 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 msgr2=0x7fb2301080f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fb2180117e0 tx=0x7fb218011ba0 comp rx=0 tx=0).stop 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 shutdown_connections 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7fb21c038f20 0x7fb21c03b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 --2- 172.21.15.73:0/108109882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2301058e0 0x7fb2301080f0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 >> 172.21.15.73:0/108109882 conn(0x7fb2301011d0 msgr2=0x7fb230101d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 shutdown_connections 2024-03-08T23:56:57.668 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:56.964+0000 7fb2357ca700 1 -- 172.21.15.73:0/108109882 wait complete. 2024-03-08T23:56:58.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/108109882' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-08T23:56:58.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:57 smithi073 ceph-mon[32309]: mgrmap e9: x(active, since 20s) 2024-03-08T23:56:58.527 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.290+0000 7fb435ac6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:56:58.527 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.389+0000 7fb435ac6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:56:58.814 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.534+0000 7fb435ac6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:56:58.814 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.632+0000 7fb435ac6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:56:58.814 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.718+0000 7fb435ac6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:56:59.121 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2726823801' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-08T23:56:59.121 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:58.996+0000 7fb435ac6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:56:59.121 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:59.125+0000 7fb435ac6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:56:59.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:56:59.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-08T23:56:59.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-08T23:56:59.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-08T23:56:59.203 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.071+0000 7f1425b03700 1 Processor -- start 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.071+0000 7f1425b03700 1 -- start start 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f1425b03700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f1425b03700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142007ade0 con 0x7f1420108130 2024-03-08T23:56:59.204 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56506/0 (socket says 172.21.15.73:56506) 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141f7fe700 1 -- 172.21.15.73:0/3064128118 learned_addr learned my addr 172.21.15.73:0/3064128118 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141f7fe700 1 -- 172.21.15.73:0/3064128118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142010aa60 con 0x7f1420108130 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141f7fe700 1 --2- 172.21.15.73:0/3064128118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14100098a0 tx=0x7f1410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d87d6a94c1ce7ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141e7fc700 1 -- 172.21.15.73:0/3064128118 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410004160 con 0x7f1420108130 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141e7fc700 1 -- 172.21.15.73:0/3064128118 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1410004300 con 0x7f1420108130 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.072+0000 7f141e7fc700 1 -- 172.21.15.73:0/3064128118 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410011b20 con 0x7f1420108130 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 -- 172.21.15.73:0/3064128118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 msgr2=0x7f142010a520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 --2- 172.21.15.73:0/3064128118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f14100098a0 tx=0x7f1410009bb0 comp rx=0 tx=0).stop 2024-03-08T23:56:59.205 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 -- 172.21.15.73:0/3064128118 shutdown_connections 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 --2- 172.21.15.73:0/3064128118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142010a520 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 -- 172.21.15.73:0/3064128118 >> 172.21.15.73:0/3064128118 conn(0x7f1420101aa0 msgr2=0x7f1420103ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 -- 172.21.15.73:0/3064128118 shutdown_connections 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.073+0000 7f1425b03700 1 -- 172.21.15.73:0/3064128118 wait complete. 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f1425b03700 1 Processor -- start 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f1425b03700 1 -- start start 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f1425b03700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:59.206 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f1425b03700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f142010ba10 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f141f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56520/0 (socket says 172.21.15.73:56520) 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.074+0000 7f141f7fe700 1 -- 172.21.15.73:0/2726823801 learned_addr learned my addr 172.21.15.73:0/2726823801 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f141f7fe700 1 -- 172.21.15.73:0/2726823801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142019bab0 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f141f7fe700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14100117e0 tx=0x7f1410011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410003bf0 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f141001a910 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142019bd40 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f142019c1a0 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.075+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410024970 con 0x7f1420108130 2024-03-08T23:56:59.207 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f1410022020 con 0x7f1420108130 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1420063410 con 0x7f1420108130 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f141cff9700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 0x7f140803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f141004d230 con 0x7f1420108130 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f141effd700 1 -- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 msgr2=0x7f140803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.076+0000 7f141effd700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 0x7f140803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.081+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1410011e20 con 0x7f1420108130 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.277+0000 7f141effd700 1 -- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 msgr2=0x7f140803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:56:59.208 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.277+0000 7f141effd700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 0x7f140803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.343+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f142019eb50 con 0x7f1420108130 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.343+0000 7f141cff9700 1 -- 172.21.15.73:0/2726823801 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7f141000ba00 con 0x7f1420108130 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.345+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 msgr2=0x7f140803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.345+0000 7f1425b03700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 0x7f140803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.345+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 msgr2=0x7f142019b570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.345+0000 7f1425b03700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f14100117e0 tx=0x7f1410011c10 comp rx=0 tx=0).stop 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 shutdown_connections 2024-03-08T23:56:59.209 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f1408038f20 0x7f140803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 --2- 172.21.15.73:0/2726823801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1420108130 0x7f142019b570 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 >> 172.21.15.73:0/2726823801 conn(0x7f1420101aa0 msgr2=0x7f1420102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 shutdown_connections 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:58.346+0000 7f1425b03700 1 -- 172.21.15.73:0/2726823801 wait complete. 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for the mgr to restart... 2024-03-08T23:56:59.210 INFO:teuthology.orchestra.run.smithi073.stdout:Waiting for mgr epoch 9... 2024-03-08T23:56:59.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:59.215+0000 7fb435ac6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:56:59.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:59.318+0000 7fb435ac6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:56:59.704 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:56:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:56:59.500+0000 7fb435ac6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:57:00.903 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:00.463+0000 7fb435ac6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:57:00.903 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:00.592+0000 7fb435ac6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:57:01.201 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:00.920+0000 7fb435ac6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:57:01.201 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:01.009+0000 7fb435ac6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:57:01.201 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:01.205+0000 7fb435ac6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:57:01.657 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:01.389+0000 7fb435ac6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:57:01.924 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:01.673+0000 7fb435ac6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:57:01.924 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:01.928+0000 7fb435ac6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:57:02.402 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:02.163+0000 7fb435ac6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:57:03.152 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:02.952+0000 7fb435ac6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:57:03.813 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:03.511+0000 7fb435ac6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:57:03.813 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:03.606+0000 7fb435ac6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:57:03.813 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-08T23:57:03.817+0000 7fb435ac6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:57:04.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: Active manager daemon x restarted 2024-03-08T23:57:04.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: Activating manager daemon x 2024-03-08T23:57:04.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: osdmap e3: 0 total, 0 up, 0 in 2024-03-08T23:57:04.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: mgrmap e10: x(active, starting, since 0.00307822s) 2024-03-08T23:57:04.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: Manager daemon x is now available 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:04.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:05.040 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-08T23:57:05.040 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-08T23:57:05.040 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:04 smithi073 ceph-mon[32309]: mgrmap e11: x(active, since 1.00566s) 2024-03-08T23:57:05.778 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout { 2024-03-08T23:57:05.778 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-08T23:57:05.778 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-08T23:57:05.778 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout } 2024-03-08T23:57:05.778 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872efcf700 1 Processor -- start 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872efcf700 1 -- start start 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872efcf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872efcf700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8728108050 con 0x7f8728107660 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872cd6b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872cd6b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52478/0 (socket says 172.21.15.73:52478) 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872cd6b700 1 -- 172.21.15.73:0/3506881404 learned_addr learned my addr 172.21.15.73:0/3506881404 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.807+0000 7f872cd6b700 1 -- 172.21.15.73:0/3506881404 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728108810 con 0x7f8728107660 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872cd6b700 1 --2- 172.21.15.73:0/3506881404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f871c0098a0 tx=0x7f871c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97749bf87387a63b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f87277fe700 1 -- 172.21.15.73:0/3506881404 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f871c004160 con 0x7f8728107660 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f87277fe700 1 -- 172.21.15.73:0/3506881404 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f871c004300 con 0x7f8728107660 2024-03-08T23:57:05.779 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f87277fe700 1 -- 172.21.15.73:0/3506881404 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f871c011b20 con 0x7f8728107660 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 -- 172.21.15.73:0/3506881404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 msgr2=0x7f8728107a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 --2- 172.21.15.73:0/3506881404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f871c0098a0 tx=0x7f871c009bb0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 -- 172.21.15.73:0/3506881404 shutdown_connections 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 --2- 172.21.15.73:0/3506881404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f8728107a80 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 -- 172.21.15.73:0/3506881404 >> 172.21.15.73:0/3506881404 conn(0x7f8728102c40 msgr2=0x7f8728105060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 -- 172.21.15.73:0/3506881404 shutdown_connections 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.808+0000 7f872efcf700 1 -- 172.21.15.73:0/3506881404 wait complete. 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872efcf700 1 Processor -- start 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872efcf700 1 -- start start 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872efcf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872efcf700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810bbc0 con 0x7f8728107660 2024-03-08T23:57:05.780 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872cd6b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872cd6b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52480/0 (socket says 172.21.15.73:52480) 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.809+0000 7f872cd6b700 1 -- 172.21.15.73:0/1793386485 learned_addr learned my addr 172.21.15.73:0/1793386485 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f872cd6b700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872819bb00 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f872cd6b700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f871c0116a0 tx=0x7f871c011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f871c003a40 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872819bd90 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f871c003be0 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f871c01a720 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.810+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872819c1f0 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.811+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f871c01aa20 con 0x7f8728107660 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.811+0000 7f8725ffb700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:05.781 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.811+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f871803baf0 con 0x7f8718038f20 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.811+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f871c04d6a0 con 0x7f8728107660 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.811+0000 7f8727fff700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:56:59.812+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:00.013+0000 7f8727fff700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:00.013+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:00.414+0000 7f8727fff700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:00.414+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:01.215+0000 7f8727fff700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:01.215+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:02.816+0000 7f8727fff700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/4041937587 2024-03-08T23:57:05.782 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:02.816+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:03.832+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f871c02d910 con 0x7f8728107660 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:03.832+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 msgr2=0x7f871803b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:03.832+0000 7f8725ffb700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.833+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f871c04f340 con 0x7f8728107660 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.834+0000 7f8725ffb700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 0x7f871803ed90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.834+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f871c02ca80 con 0x7f871803c9a0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.834+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 0x7f871803ed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.835+0000 7f8727fff700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 0x7f871803ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8714003a10 tx=0x7f87140092b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.835+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f871c02ca80 con 0x7f871803c9a0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f8728063410 con 0x7f871803c9a0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f8725ffb700 1 -- 172.21.15.73:0/1793386485 <== mgr.14152 v2:172.21.15.73:6800/2701889057 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f8728063410 con 0x7f871803c9a0 2024-03-08T23:57:05.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 msgr2=0x7f871803ed90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 0x7f871803ed90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f8714003a10 tx=0x7f87140092b0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 msgr2=0x7f872819b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f871c0116a0 tx=0x7f871c011ba0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 shutdown_connections 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f871803c9a0 0x7f871803ed90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728107660 0x7f872819b5c0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 --2- 172.21.15.73:0/1793386485 >> [v2:172.21.15.73:6800/4041937587,v1:172.21.15.73:6801/4041937587] conn(0x7f8718038f20 0x7f871803b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 >> 172.21.15.73:0/1793386485 conn(0x7f8728102c40 msgr2=0x7f8728103710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 shutdown_connections 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:04.842+0000 7f872efcf700 1 -- 172.21.15.73:0/1793386485 wait complete. 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:mgr epoch 9 is available 2024-03-08T23:57:05.784 INFO:teuthology.orchestra.run.smithi073.stdout:Generating a dashboard self-signed certificate... 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: [08/Mar/2024:23:57:04] ENGINE Bus STARTING 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: [08/Mar/2024:23:57:04] ENGINE Serving on http://172.21.15.73:8765 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: [08/Mar/2024:23:57:04] ENGINE Serving on https://172.21.15.73:7150 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: [08/Mar/2024:23:57:04] ENGINE Bus STARTED 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-08T23:57:06.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:07.005 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.211+0000 7f64c8d9e700 1 Processor -- start 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.211+0000 7f64c8d9e700 1 -- start start 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.211+0000 7f64c8d9e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.211+0000 7f64c8d9e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c4106d70 con 0x7f64c4106380 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52572/0 (socket says 172.21.15.73:52572) 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c37fe700 1 -- 172.21.15.73:0/2921950741 learned_addr learned my addr 172.21.15.73:0/2921950741 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c37fe700 1 -- 172.21.15.73:0/2921950741 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c4107580 con 0x7f64c4106380 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c37fe700 1 --2- 172.21.15.73:0/2921950741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f64b40098a0 tx=0x7f64b4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b0055aa39f2118 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c27fc700 1 -- 172.21.15.73:0/2921950741 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64b4004160 con 0x7f64c4106380 2024-03-08T23:57:07.006 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c27fc700 1 -- 172.21.15.73:0/2921950741 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64b4004300 con 0x7f64c4106380 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.212+0000 7f64c27fc700 1 -- 172.21.15.73:0/2921950741 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64b4011b20 con 0x7f64c4106380 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 -- 172.21.15.73:0/2921950741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 msgr2=0x7f64c41067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/2921950741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f64b40098a0 tx=0x7f64b4009bb0 comp rx=0 tx=0).stop 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 -- 172.21.15.73:0/2921950741 shutdown_connections 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/2921950741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c41067a0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 -- 172.21.15.73:0/2921950741 >> 172.21.15.73:0/2921950741 conn(0x7f64c4101960 msgr2=0x7f64c4103d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 -- 172.21.15.73:0/2921950741 shutdown_connections 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.213+0000 7f64c8d9e700 1 -- 172.21.15.73:0/2921950741 wait complete. 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c8d9e700 1 Processor -- start 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c8d9e700 1 -- start start 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c8d9e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c8d9e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64c410a930 con 0x7f64c4106380 2024-03-08T23:57:07.007 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c37fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52588/0 (socket says 172.21.15.73:52588) 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.214+0000 7f64c37fe700 1 -- 172.21.15.73:0/1626163211 learned_addr learned my addr 172.21.15.73:0/1626163211 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c37fe700 1 -- 172.21.15.73:0/1626163211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c419e140 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c37fe700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f64b40116a0 tx=0x7f64b4011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64b4003a40 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64b4003be0 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64c419e310 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f64b401a720 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.215+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64c419e6b0 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.216+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f64b401aaa0 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.216+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64c4063410 con 0x7f64c4106380 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.216+0000 7f64c0ff9700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 0x7f64ac03b2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:07.008 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.216+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f64b404d3e0 con 0x7f64c4106380 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.217+0000 7f64c2ffd700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 0x7f64ac03b2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.217+0000 7f64c2ffd700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 0x7f64ac03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64b8006fd0 tx=0x7f64b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.222+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f64b4011e20 con 0x7f64c4106380 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.403+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f64b402d910 con 0x7f64c4106380 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.432+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f64c4102b40 con 0x7f64ac038e10 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.503+0000 7f64c0ff9700 1 -- 172.21.15.73:0/1626163211 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f64c4102b40 con 0x7f64ac038e10 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 msgr2=0x7f64ac03b2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 0x7f64ac03b2d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f64b8006fd0 tx=0x7f64b8006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 msgr2=0x7f64c407aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f64b40116a0 tx=0x7f64b4011ba0 comp rx=0 tx=0).stop 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 shutdown_connections 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f64ac038e10 0x7f64ac03b2d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:07.009 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 --2- 172.21.15.73:0/1626163211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f64c4106380 0x7f64c407aa60 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:07.010 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 >> 172.21.15.73:0/1626163211 conn(0x7f64c4101960 msgr2=0x7f64c4102430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:07.010 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 shutdown_connections 2024-03-08T23:57:07.010 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:06.505+0000 7f64c8d9e700 1 -- 172.21.15.73:0/1626163211 wait complete. 2024-03-08T23:57:07.010 INFO:teuthology.orchestra.run.smithi073.stdout:Creating initial admin user... 2024-03-08T23:57:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:07 smithi073 ceph-mon[32309]: mgrmap e12: x(active, since 2s) 2024-03-08T23:57:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:07 smithi073 ceph-mon[32309]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$uHLCfOVWYRzfh/z0BKJLn.qTWGkiOlyN1jZqdYqImLMbLuU/9gd.C", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1709942227, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d2bec700 1 Processor -- start 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d2bec700 1 -- start start 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d2bec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d2bec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc106f10 con 0x7f13cc106520 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d0988700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d0988700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52596/0 (socket says 172.21.15.73:52596) 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.427+0000 7f13d0988700 1 -- 172.21.15.73:0/509769110 learned_addr learned my addr 172.21.15.73:0/509769110 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.428+0000 7f13d0988700 1 -- 172.21.15.73:0/509769110 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc107720 con 0x7f13cc106520 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.428+0000 7f13d0988700 1 --2- 172.21.15.73:0/509769110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f13c00098a0 tx=0x7f13c0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=229650006a63c6c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:08.783 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.428+0000 7f13cb7fe700 1 -- 172.21.15.73:0/509769110 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13c0004160 con 0x7f13cc106520 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.428+0000 7f13cb7fe700 1 -- 172.21.15.73:0/509769110 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13c0004300 con 0x7f13cc106520 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.428+0000 7f13cb7fe700 1 -- 172.21.15.73:0/509769110 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13c0011b20 con 0x7f13cc106520 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- 172.21.15.73:0/509769110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 msgr2=0x7f13cc106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 --2- 172.21.15.73:0/509769110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f13c00098a0 tx=0x7f13c0009bb0 comp rx=0 tx=0).stop 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- 172.21.15.73:0/509769110 shutdown_connections 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 --2- 172.21.15.73:0/509769110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc106940 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- 172.21.15.73:0/509769110 >> 172.21.15.73:0/509769110 conn(0x7f13cc101aa0 msgr2=0x7f13cc103f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- 172.21.15.73:0/509769110 shutdown_connections 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- 172.21.15.73:0/509769110 wait complete. 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 Processor -- start 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- start start 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.429+0000 7f13d2bec700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13cc10aad0 con 0x7f13cc106520 2024-03-08T23:57:08.784 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d0988700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d0988700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52602/0 (socket says 172.21.15.73:52602) 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d0988700 1 -- 172.21.15.73:0/108602791 learned_addr learned my addr 172.21.15.73:0/108602791 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d0988700 1 -- 172.21.15.73:0/108602791 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc19e320 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d0988700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13c0000c00 tx=0x7f13c0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13c0003b60 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f13c001a910 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13cc19e4f0 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f13c0024930 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.430+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13cc19e890 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.431+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f13c0022020 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.431+0000 7f13c9ffb700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 0x7f13bc03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.431+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13cc063410 con 0x7f13cc106520 2024-03-08T23:57:08.785 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.432+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f13c004d1d0 con 0x7f13cc106520 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.432+0000 7f13cbfff700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 0x7f13bc03b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.432+0000 7f13cbfff700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 0x7f13bc03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13b8006fd0 tx=0x7f13b8006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.436+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f13c002d3f0 con 0x7f13cc106520 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.666+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f13cc19ec60 con 0x7f13bc038f70 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.891+0000 7f13c9ffb700 1 -- 172.21.15.73:0/108602791 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f13cc19ec60 con 0x7f13bc038f70 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.893+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 msgr2=0x7f13bc03b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.893+0000 7f13d2bec700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 0x7f13bc03b430 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f13b8006fd0 tx=0x7f13b8006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.893+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 msgr2=0x7f13cc07aa60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.893+0000 7f13d2bec700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f13c0000c00 tx=0x7f13c0011c10 comp rx=0 tx=0).stop 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.893+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 shutdown_connections 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.894+0000 7f13d2bec700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13bc038f70 0x7f13bc03b430 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.894+0000 7f13d2bec700 1 --2- 172.21.15.73:0/108602791 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f13cc106520 0x7f13cc07aa60 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.894+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 >> 172.21.15.73:0/108602791 conn(0x7f13cc101aa0 msgr2=0x7f13cc102600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:08.786 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.894+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 shutdown_connections 2024-03-08T23:57:08.787 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:07.894+0000 7f13d2bec700 1 -- 172.21.15.73:0/108602791 wait complete. 2024-03-08T23:57:08.787 INFO:teuthology.orchestra.run.smithi073.stdout:Fetching dashboard port number... 2024-03-08T23:57:09.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:08 smithi073 ceph-mon[32309]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:09.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:10.134 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:09 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3923607437' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-08T23:57:10.400 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stdout 8443 2024-03-08T23:57:10.400 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.634+0000 7fdfc4632700 1 Processor -- start 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.634+0000 7fdfc4632700 1 -- start start 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.634+0000 7fdfc4632700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.634+0000 7fdfc4632700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc106f40 con 0x7fdfbc106550 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.634+0000 7fdfc23ce700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc23ce700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41414/0 (socket says 172.21.15.73:41414) 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc23ce700 1 -- 172.21.15.73:0/4056121986 learned_addr learned my addr 172.21.15.73:0/4056121986 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc23ce700 1 -- 172.21.15.73:0/4056121986 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc107750 con 0x7fdfbc106550 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc23ce700 1 --2- 172.21.15.73:0/4056121986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800bb40 tx=0x7fdfb800be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1616d7a612f4a04b server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc13cc700 1 -- 172.21.15.73:0/4056121986 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfb800dbe0 con 0x7fdfbc106550 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc13cc700 1 -- 172.21.15.73:0/4056121986 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfb8004510 con 0x7fdfbc106550 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.635+0000 7fdfc13cc700 1 -- 172.21.15.73:0/4056121986 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfb8010470 con 0x7fdfbc106550 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- 172.21.15.73:0/4056121986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 msgr2=0x7fdfbc106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 --2- 172.21.15.73:0/4056121986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fdfb800bb40 tx=0x7fdfb800be50 comp rx=0 tx=0).stop 2024-03-08T23:57:10.401 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- 172.21.15.73:0/4056121986 shutdown_connections 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 --2- 172.21.15.73:0/4056121986 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc106970 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- 172.21.15.73:0/4056121986 >> 172.21.15.73:0/4056121986 conn(0x7fdfbc101b70 msgr2=0x7fdfbc103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- 172.21.15.73:0/4056121986 shutdown_connections 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- 172.21.15.73:0/4056121986 wait complete. 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 Processor -- start 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.636+0000 7fdfc4632700 1 -- start start 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc4632700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc4632700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfbc079fc0 con 0x7fdfbc106550 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc23ce700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc23ce700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41426/0 (socket says 172.21.15.73:41426) 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc23ce700 1 -- 172.21.15.73:0/3923607437 learned_addr learned my addr 172.21.15.73:0/3923607437 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:10.402 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc23ce700 1 -- 172.21.15.73:0/3923607437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfbc199970 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc23ce700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80046e0 tx=0x7fdfb800ded0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfb8027070 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdfb8004350 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfbc199c00 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdfb8022a60 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.637+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfbc19c8f0 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.638+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fdfb802a070 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.638+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfbc063410 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.638+0000 7fdfb37fe700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 0x7fdfa803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.638+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdfb801f070 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.638+0000 7fdfc1bcd700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 0x7fdfa803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.639+0000 7fdfc1bcd700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 0x7fdfa803b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfac006fd0 tx=0x7fdfac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.643+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdfb8015070 con 0x7fdfbc106550 2024-03-08T23:57:10.403 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.842+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fdfbc192990 con 0x7fdfbc106550 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.842+0000 7fdfb37fe700 1 -- 172.21.15.73:0/3923607437 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7fdfbc192990 con 0x7fdfbc106550 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.844+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 msgr2=0x7fdfa803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.844+0000 7fdfc4632700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 0x7fdfa803b430 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fdfac006fd0 tx=0x7fdfac006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.844+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 msgr2=0x7fdfbc199430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.844+0000 7fdfc4632700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fdfb80046e0 tx=0x7fdfb800ded0 comp rx=0 tx=0).stop 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 shutdown_connections 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdfa8038f70 0x7fdfa803b430 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 --2- 172.21.15.73:0/3923607437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdfbc106550 0x7fdfbc199430 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 >> 172.21.15.73:0/3923607437 conn(0x7fdfbc101b70 msgr2=0x7fdfbc18fa70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 shutdown_connections 2024-03-08T23:57:10.404 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:09.845+0000 7fdfc4632700 1 -- 172.21.15.73:0/3923607437 wait complete. 2024-03-08T23:57:10.406 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-08T23:57:10.406 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout disabled 2024-03-08T23:57:10.413 INFO:teuthology.orchestra.run.smithi073.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-08T23:57:10.413 INFO:teuthology.orchestra.run.smithi073.stdout:systemctl: stdout inactive 2024-03-08T23:57:10.413 INFO:teuthology.orchestra.run.smithi073.stdout:firewalld.service is not enabled 2024-03-08T23:57:10.413 INFO:teuthology.orchestra.run.smithi073.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout:Ceph Dashboard is now available at: 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout: URL: https://smithi073.front.sepia.ceph.com:8443/ 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout: User: admin 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout: Password: gkird06kmt 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:10.415 INFO:teuthology.orchestra.run.smithi073.stdout:Saving cluster configuration to /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config directory 2024-03-08T23:57:10.416 INFO:teuthology.orchestra.run.smithi073.stdout:Enabling autotune for osd_memory_target 2024-03-08T23:57:11.053 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:10 smithi073 ceph-mon[32309]: mgrmap e13: x(active, since 6s) 2024-03-08T23:57:11.708 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.851+0000 7f872d394700 1 Processor -- start 2024-03-08T23:57:11.708 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.851+0000 7f872d394700 1 -- start start 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.851+0000 7f872d394700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.851+0000 7f872d394700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8728106f20 con 0x7f8728106530 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41436/0 (socket says 172.21.15.73:41436) 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f8726ffd700 1 -- 172.21.15.73:0/3426662089 learned_addr learned my addr 172.21.15.73:0/3426662089 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f8726ffd700 1 -- 172.21.15.73:0/3426662089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8728107730 con 0x7f8728106530 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f8726ffd700 1 --2- 172.21.15.73:0/3426662089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87180098a0 tx=0x7f8718009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=14273cbc676c5a50 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f87267fc700 1 -- 172.21.15.73:0/3426662089 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8718004160 con 0x7f8728106530 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f87267fc700 1 -- 172.21.15.73:0/3426662089 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718004300 con 0x7f8728106530 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.852+0000 7f87267fc700 1 -- 172.21.15.73:0/3426662089 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8718011b20 con 0x7f8728106530 2024-03-08T23:57:11.709 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 -- 172.21.15.73:0/3426662089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 msgr2=0x7f8728106950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 --2- 172.21.15.73:0/3426662089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f87180098a0 tx=0x7f8718009bb0 comp rx=0 tx=0).stop 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 -- 172.21.15.73:0/3426662089 shutdown_connections 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 --2- 172.21.15.73:0/3426662089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f8728106950 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 -- 172.21.15.73:0/3426662089 >> 172.21.15.73:0/3426662089 conn(0x7f8728101b70 msgr2=0x7f8728103f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 -- 172.21.15.73:0/3426662089 shutdown_connections 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 -- 172.21.15.73:0/3426662089 wait complete. 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.853+0000 7f872d394700 1 Processor -- start 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f872d394700 1 -- start start 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f872d394700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f872d394700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f872810aae0 con 0x7f8728106530 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:11.710 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8726ffd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41452/0 (socket says 172.21.15.73:41452) 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8726ffd700 1 -- 172.21.15.73:0/2649728778 learned_addr learned my addr 172.21.15.73:0/2649728778 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8726ffd700 1 -- 172.21.15.73:0/2649728778 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f872819e350 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8726ffd700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87180116a0 tx=0x7f8718011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8718003a40 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8718003be0 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f872819e520 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f871801a720 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.854+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f872819e8c0 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.855+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f871801a9d0 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.855+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8728063410 con 0x7f8728106530 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.855+0000 7f8724ff9700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 0x7f871403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:11.711 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.855+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f871804d6a0 con 0x7f8728106530 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.856+0000 7f871ffff700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 0x7f871403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.856+0000 7f871ffff700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 0x7f871403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8710006fd0 tx=0x7f8710006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:10.860+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8718011e20 con 0x7f8728106530 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.056+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f872819ecd0 con 0x7f8728106530 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.057+0000 7f8724ff9700 1 -- 172.21.15.73:0/2649728778 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7f871801adc0 con 0x7f8728106530 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 msgr2=0x7f871403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 0x7f871403b4e0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8710006fd0 tx=0x7f8710006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 msgr2=0x7f872807aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f87180116a0 tx=0x7f8718011ba0 comp rx=0 tx=0).stop 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 shutdown_connections 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8714039020 0x7f871403b4e0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:11.712 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 --2- 172.21.15.73:0/2649728778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8728106530 0x7f872807aa40 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:11.713 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 >> 172.21.15.73:0/2649728778 conn(0x7f8728101b70 msgr2=0x7f8728102680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:11.713 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 shutdown_connections 2024-03-08T23:57:11.713 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:11.059+0000 7f872d394700 1 -- 172.21.15.73:0/2649728778 wait complete. 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.145+0000 7efe3b9b1700 1 Processor -- start 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.145+0000 7efe3b9b1700 1 -- start start 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3b9b1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3b9b1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3407cff0 con 0x7efe3407b310 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3974d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3974d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41458/0 (socket says 172.21.15.73:41458) 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3974d700 1 -- 172.21.15.73:0/3377511476 learned_addr learned my addr 172.21.15.73:0/3377511476 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:13.048 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3974d700 1 -- 172.21.15.73:0/3377511476 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3407bc70 con 0x7efe3407b310 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe3974d700 1 --2- 172.21.15.73:0/3377511476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe24009e80 tx=0x7efe2400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=50ed92c5f6c918d8 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe33fff700 1 -- 172.21.15.73:0/3377511476 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe24017070 con 0x7efe3407b310 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe33fff700 1 -- 172.21.15.73:0/3377511476 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe2400b8c0 con 0x7efe3407b310 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.146+0000 7efe33fff700 1 -- 172.21.15.73:0/3377511476 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe240045c0 con 0x7efe3407b310 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 -- 172.21.15.73:0/3377511476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 msgr2=0x7efe3407b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/3377511476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7efe24009e80 tx=0x7efe2400b330 comp rx=0 tx=0).stop 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 -- 172.21.15.73:0/3377511476 shutdown_connections 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/3377511476 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3407b730 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 -- 172.21.15.73:0/3377511476 >> 172.21.15.73:0/3377511476 conn(0x7efe34102c10 msgr2=0x7efe34105030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 -- 172.21.15.73:0/3377511476 shutdown_connections 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.147+0000 7efe3b9b1700 1 -- 172.21.15.73:0/3377511476 wait complete. 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3b9b1700 1 Processor -- start 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3b9b1700 1 -- start start 2024-03-08T23:57:13.049 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3b9b1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3b9b1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe3410dd10 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3974d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3974d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41472/0 (socket says 172.21.15.73:41472) 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3974d700 1 -- 172.21.15.73:0/2089798740 learned_addr learned my addr 172.21.15.73:0/2089798740 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.148+0000 7efe3974d700 1 -- 172.21.15.73:0/2089798740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe3419dd80 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe3974d700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe24009fd0 tx=0x7efe24003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe24020020 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efe2400bba0 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe3419e010 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7efe24017040 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.149+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe341a0d00 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7efe24023020 con 0x7efe3407b310 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe327fc700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 0x7efe1c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:13.050 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7efe2400f050 con 0x7efe3407b310 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe34063410 con 0x7efe3407b310 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe38f4c700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 0x7efe1c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.150+0000 7efe38f4c700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 0x7efe1c03b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe28006fd0 tx=0x7efe28006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.155+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efe2402a020 con 0x7efe3407b310 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.462+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7efe341a10e0 con 0x7efe3407b310 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.464+0000 7efe327fc700 1 -- 172.21.15.73:0/2089798740 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7efe2402c560 con 0x7efe3407b310 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 msgr2=0x7efe1c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 0x7efe1c03b530 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7efe28006fd0 tx=0x7efe28006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 msgr2=0x7efe3419d840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7efe24009fd0 tx=0x7efe24003d00 comp rx=0 tx=0).stop 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 shutdown_connections 2024-03-08T23:57:13.051 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efe1c039070 0x7efe1c03b530 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 --2- 172.21.15.73:0/2089798740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efe3407b310 0x7efe3419d840 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 >> 172.21.15.73:0/2089798740 conn(0x7efe34102c10 msgr2=0x7efe34103720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 shutdown_connections 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr 2024-03-08T23:57:12.467+0000 7efe3b9b1700 1 -- 172.21.15.73:0/2089798740 wait complete. 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:Or, if you are only running a single cluster on this host: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-08T23:57:13.052 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout: ceph telemetry on 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout:For more information see: 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:13.053 INFO:teuthology.orchestra.run.smithi073.stdout:Bootstrap complete. 2024-03-08T23:57:13.088 INFO:tasks.cephadm:Fetching config... 2024-03-08T23:57:13.088 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:57:13.088 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-08T23:57:13.105 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-08T23:57:13.105 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:57:13.105 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-08T23:57:13.161 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-08T23:57:13.161 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:57:13.161 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-08T23:57:13.228 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-08T23:57:13.228 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:57:13.228 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-08T23:57:13.283 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-08T23:57:13.284 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T23:57:13.366 INFO:teuthology.orchestra.run.smithi073.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:57:13.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T23:57:13.424 INFO:teuthology.orchestra.run.smithi102.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:57:13.439 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-08T23:57:13.484 INFO:teuthology.orchestra.run.smithi179.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCeO025a45gNXB0a8KY462gfe9JNXfkKzebeYugitjLsVixGAuvrIj4UyhdrSLfKs3NVYzS5CsvP64bvE10tc3MJ/lTQstInEL+aRLibzVtpNN9XruCF8v9IcFNAjtJjUREoioqb4XX97OmdSKUOW/LJ9vV6pcX1v3PQpjE3txb5WOlT7iyiuuEtH5IOj7CRixsq3KZSwH3a44dLuOFeYTLv6e8iRD7dyAN7UkQriTMwqqhGbvli7RJb8H/1KgiSufR/N1/pcW4S/hAlet5FCNhMihIG6CkoLYr6AKRD2Mvl14+m7F+dZWj+euyG3Ju1TnkPkG5qdLoeRMB3bAYjUVpG0RKE9doiUry0/Q4UI7QLXqQDTWJPxMP5coy750cXEFXDdVf7qczUflxThBllx6b1dnSq3beAs+PxadGMMxpk3gdJ/2ylpuzT11hSICNJVjctJUKHyGq1N4OSBSmxHcakWWdn7yBKsvczDdXfZywuvBjTxWqK+1flozJS8C/sGk= ceph-44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:57:13.499 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-08T23:57:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:13 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2089798740' entity='client.admin' 2024-03-08T23:57:13.739 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi073", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:16.221 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:16.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 -- 172.21.15.73:0/963223067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 msgr2=0x7f8200100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 --2- 172.21.15.73:0/963223067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200100b30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f81f8009910 tx=0x7f81f8009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 -- 172.21.15.73:0/963223067 shutdown_connections 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 --2- 172.21.15.73:0/963223067 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200100b30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 -- 172.21.15.73:0/963223067 >> 172.21.15.73:0/963223067 conn(0x7f82000fbcb0 msgr2=0x7f82000fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 -- 172.21.15.73:0/963223067 shutdown_connections 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.329+0000 7f8204937700 1 -- 172.21.15.73:0/963223067 wait complete. 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.330+0000 7f8204937700 1 Processor -- start 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.330+0000 7f8204937700 1 -- start start 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.330+0000 7f8204937700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f8204937700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82001067a0 con 0x7f8200100710 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81fe59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:16.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81fe59c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41500/0 (socket says 172.21.15.73:41500) 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81fe59c700 1 -- 172.21.15.73:0/2952816725 learned_addr learned my addr 172.21.15.73:0/2952816725 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81fe59c700 1 -- 172.21.15.73:0/2952816725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8200199c00 con 0x7f8200100710 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81fe59c700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f81f8006010 tx=0x7f81f8005050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f8027070 con 0x7f8200100710 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81f800be40 con 0x7f8200100710 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f81f800f460 con 0x7f8200100710 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8200199dd0 con 0x7f8200100710 2024-03-08T23:57:16.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.331+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f820019a1f0 con 0x7f8200100710 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.332+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f81f802a070 con 0x7f8200100710 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.333+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8200106f20 con 0x7f8200100710 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.333+0000 7f81ef7fe700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 0x7f81e803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.333+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f81f801e030 con 0x7f8200100710 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.333+0000 7f81fdd9b700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 0x7f81e803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:16.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.333+0000 7f81fdd9b700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 0x7f81e803b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:16.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.338+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f81f8013070 con 0x7f8200100710 2024-03-08T23:57:16.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.541+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f820002da60 con 0x7f8200100710 2024-03-08T23:57:16.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.543+0000 7f81ef7fe700 1 -- 172.21.15.73:0/2952816725 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f81f802a360 con 0x7f8200100710 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 msgr2=0x7f81e803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 0x7f81e803b480 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f81f4006fd0 tx=0x7f81f4006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 msgr2=0x7f8200069cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f81f8006010 tx=0x7f81f8005050 comp rx=0 tx=0).stop 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 shutdown_connections 2024-03-08T23:57:16.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f81e8038fc0 0x7f81e803b480 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:16.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 --2- 172.21.15.73:0/2952816725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8200100710 0x7f8200069cb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:16.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 >> 172.21.15.73:0/2952816725 conn(0x7f82000fbcb0 msgr2=0x7f82000fc700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:16.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 shutdown_connections 2024-03-08T23:57:16.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:16.546+0000 7f8204937700 1 -- 172.21.15.73:0/2952816725 wait complete. 2024-03-08T23:57:17.319 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-08T23:57:17.319 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-08T23:57:17.586 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:17 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2952816725' entity='client.admin' 2024-03-08T23:57:17.586 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:17.586 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:17.788 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- 172.21.15.73:0/2916020346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 msgr2=0x7fb3b8100b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/2916020346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8100b30 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac009910 tx=0x7fb3ac009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- 172.21.15.73:0/2916020346 shutdown_connections 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/2916020346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8100b30 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- 172.21.15.73:0/2916020346 >> 172.21.15.73:0/2916020346 conn(0x7fb3b80fbcb0 msgr2=0x7fb3b80fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- 172.21.15.73:0/2916020346 shutdown_connections 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- 172.21.15.73:0/2916020346 wait complete. 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 Processor -- start 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.388+0000 7fb3bebd5700 1 -- start start 2024-03-08T23:57:19.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bebd5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bebd5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b81067a0 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bc971700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bc971700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34070/0 (socket says 172.21.15.73:34070) 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bc971700 1 -- 172.21.15.73:0/1573079966 learned_addr learned my addr 172.21.15.73:0/1573079966 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bc971700 1 -- 172.21.15.73:0/1573079966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b81974a0 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bc971700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac005850 tx=0x7fb3ac005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3ac026070 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3ac0044a0 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b8197730 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.389+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3ac003ab0 con 0x7fb3b8100710 2024-03-08T23:57:19.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.390+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b819a420 con 0x7fb3b8100710 2024-03-08T23:57:19.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.390+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb3ac029070 con 0x7fb3b8100710 2024-03-08T23:57:19.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.390+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b8106f20 con 0x7fb3b8100710 2024-03-08T23:57:19.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.390+0000 7fb3b5ffb700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 0x7fb3a803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:19.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.391+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb3ac01e070 con 0x7fb3b8100710 2024-03-08T23:57:19.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.391+0000 7fb3b7fff700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 0x7fb3a803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:19.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.391+0000 7fb3b7fff700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 0x7fb3a803b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006fd0 tx=0x7fb3a4006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:19.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.395+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb3ac052020 con 0x7fb3b8100710 2024-03-08T23:57:19.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.595+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3b802d940 con 0x7fb3a8039020 2024-03-08T23:57:19.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.598+0000 7fb3b5ffb700 1 -- 172.21.15.73:0/1573079966 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb3b802d940 con 0x7fb3a8039020 2024-03-08T23:57:19.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.600+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 msgr2=0x7fb3a803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:19.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.600+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 0x7fb3a803b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fb3a4006fd0 tx=0x7fb3a4006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.600+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 msgr2=0x7fb3b8196f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.600+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fb3ac005850 tx=0x7fb3ac005030 comp rx=0 tx=0).stop 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 shutdown_connections 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb3a8039020 0x7fb3a803b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 --2- 172.21.15.73:0/1573079966 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3b8100710 0x7fb3b8196f60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 >> 172.21.15.73:0/1573079966 conn(0x7fb3b80fbcb0 msgr2=0x7fb3b806a100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 shutdown_connections 2024-03-08T23:57:19.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:19.601+0000 7fb3bebd5700 1 -- 172.21.15.73:0/1573079966 wait complete. 2024-03-08T23:57:20.388 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi102 2024-03-08T23:57:20.388 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:57:20.388 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.conf 2024-03-08T23:57:20.406 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:57:20.406 DEBUG:teuthology.orchestra.run.smithi102:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:20.463 INFO:tasks.cephadm:Adding host smithi102 to orchestrator... 2024-03-08T23:57:20.463 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch host add smithi102 2024-03-08T23:57:20.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:20.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:20.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:20.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:20.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:20.702 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:21.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:21 smithi073 ceph-mon[32309]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:21.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:21 smithi073 ceph-mon[32309]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:57:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 -- 172.21.15.73:0/1869322424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 msgr2=0x7eff84106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/1869322424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff84106aa0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7eff74009910 tx=0x7eff74009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 -- 172.21.15.73:0/1869322424 shutdown_connections 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/1869322424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff84106aa0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 -- 172.21.15.73:0/1869322424 >> 172.21.15.73:0/1869322424 conn(0x7eff840fc1d0 msgr2=0x7eff840fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 -- 172.21.15.73:0/1869322424 shutdown_connections 2024-03-08T23:57:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.297+0000 7eff8b3e0700 1 -- 172.21.15.73:0/1869322424 wait complete. 2024-03-08T23:57:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8b3e0700 1 Processor -- start 2024-03-08T23:57:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8b3e0700 1 -- start start 2024-03-08T23:57:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8b3e0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8b3e0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff8410ad00 con 0x7eff841066c0 2024-03-08T23:57:22.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8917c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8917c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34086/0 (socket says 172.21.15.73:34086) 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8917c700 1 -- 172.21.15.73:0/298158096 learned_addr learned my addr 172.21.15.73:0/298158096 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.298+0000 7eff8917c700 1 -- 172.21.15.73:0/298158096 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff84197810 con 0x7eff841066c0 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff8917c700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff74005850 tx=0x7eff74005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff74026070 con 0x7eff841066c0 2024-03-08T23:57:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff7400bbd0 con 0x7eff841066c0 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff84197b00 con 0x7eff841066c0 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7eff74003970 con 0x7eff841066c0 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff8419bba0 con 0x7eff841066c0 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.299+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7eff74029070 con 0x7eff841066c0 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.300+0000 7eff7a7fc700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 0x7eff7003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.300+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7eff7401e070 con 0x7eff841066c0 2024-03-08T23:57:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.300+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff84101e00 con 0x7eff841066c0 2024-03-08T23:57:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.300+0000 7eff8897b700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 0x7eff7003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.300+0000 7eff8897b700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 0x7eff7003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.305+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7eff74013070 con 0x7eff841066c0 2024-03-08T23:57:22.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:22.501+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}) v1 -- 0x7eff84198630 con 0x7eff70039020 2024-03-08T23:57:23.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:23 smithi073 ceph-mon[32309]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi102", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:24.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:24 smithi073 ceph-mon[32309]: Deploying cephadm binary to smithi102 2024-03-08T23:57:24.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:24 smithi073 ceph-mon[32309]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:26.668 INFO:teuthology.orchestra.run.smithi073.stdout:Added host 'smithi102' with addr '172.21.15.102' 2024-03-08T23:57:26.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.672+0000 7eff7a7fc700 1 -- 172.21.15.73:0/298158096 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7eff84198630 con 0x7eff70039020 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 msgr2=0x7eff7003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 0x7eff7003b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7eff80006fd0 tx=0x7eff80006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 msgr2=0x7eff841972d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7eff74005850 tx=0x7eff74005030 comp rx=0 tx=0).stop 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 shutdown_connections 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7eff70039020 0x7eff7003b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:26.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 --2- 172.21.15.73:0/298158096 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7eff841066c0 0x7eff841972d0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:26.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 >> 172.21.15.73:0/298158096 conn(0x7eff840fc1d0 msgr2=0x7eff841041d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:26.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 shutdown_connections 2024-03-08T23:57:26.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:26.674+0000 7eff8b3e0700 1 -- 172.21.15.73:0/298158096 wait complete. 2024-03-08T23:57:26.978 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:26 smithi073 ceph-mon[32309]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:26.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:26.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:27.329 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-08T23:57:27.570 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:27.957 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:27 smithi073 ceph-mon[32309]: Added host smithi102 2024-03-08T23:57:27.957 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:27 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:27.957 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:27 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:28.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 -- 172.21.15.73:0/3799856664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 msgr2=0x7fb72c106a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:28.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 --2- 172.21.15.73:0/3799856664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c106a80 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fb71c009910 tx=0x7fb71c009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:28.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 -- 172.21.15.73:0/3799856664 shutdown_connections 2024-03-08T23:57:28.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 --2- 172.21.15.73:0/3799856664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c106a80 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:28.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 -- 172.21.15.73:0/3799856664 >> 172.21.15.73:0/3799856664 conn(0x7fb72c0fc190 msgr2=0x7fb72c0fe5b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:28.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 -- 172.21.15.73:0/3799856664 shutdown_connections 2024-03-08T23:57:28.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.981+0000 7fb732726700 1 -- 172.21.15.73:0/3799856664 wait complete. 2024-03-08T23:57:28.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.982+0000 7fb732726700 1 Processor -- start 2024-03-08T23:57:28.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.982+0000 7fb732726700 1 -- start start 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.982+0000 7fb732726700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.982+0000 7fb732726700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb72c10ace0 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb72bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb72bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46040/0 (socket says 172.21.15.73:46040) 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb72bfff700 1 -- 172.21.15.73:0/4024905830 learned_addr learned my addr 172.21.15.73:0/4024905830 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb72bfff700 1 -- 172.21.15.73:0/4024905830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb72c198680 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb72bfff700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb71c006010 tx=0x7fb71c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb71c026070 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb72c198970 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb71c00bbd0 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb71c003910 con 0x7fb72c1066a0 2024-03-08T23:57:28.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.983+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb72c198e30 con 0x7fb72c1066a0 2024-03-08T23:57:28.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.984+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb72c101de0 con 0x7fb72c1066a0 2024-03-08T23:57:28.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.984+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fb71c029070 con 0x7fb72c1066a0 2024-03-08T23:57:28.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.984+0000 7fb7297fa700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 0x7fb71403b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:28.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.985+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb71c01e070 con 0x7fb72c1066a0 2024-03-08T23:57:28.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.985+0000 7fb72b7fe700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 0x7fb71403b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:28.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.985+0000 7fb72b7fe700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 0x7fb71403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb720006fd0 tx=0x7fb720006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:28.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:28.993+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb71c013070 con 0x7fb72c1066a0 2024-03-08T23:57:29.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:28 smithi073 ceph-mon[32309]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:29.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.203+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fb72c06a050 con 0x7fb714038fc0 2024-03-08T23:57:29.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.204+0000 7fb7297fa700 1 -- 172.21.15.73:0/4024905830 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+160 (secure 0 0 0) 0x7fb72c06a050 con 0x7fb714038fc0 2024-03-08T23:57:29.200 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:29.201 INFO:teuthology.orchestra.run.smithi073.stdout:[{"addr": "172.21.15.73", "hostname": "smithi073", "labels": [], "status": ""}, {"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}] 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 msgr2=0x7fb71403b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 0x7fb71403b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fb720006fd0 tx=0x7fb720006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 msgr2=0x7fb72c198140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fb71c006010 tx=0x7fb71c005030 comp rx=0 tx=0).stop 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 shutdown_connections 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb714038fc0 0x7fb71403b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 --2- 172.21.15.73:0/4024905830 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb72c1066a0 0x7fb72c198140 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 >> 172.21.15.73:0/4024905830 conn(0x7fb72c0fc190 msgr2=0x7fb72c108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 shutdown_connections 2024-03-08T23:57:29.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:29.206+0000 7fb732726700 1 -- 172.21.15.73:0/4024905830 wait complete. 2024-03-08T23:57:29.809 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi179 2024-03-08T23:57:29.810 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:57:29.810 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.conf 2024-03-08T23:57:29.829 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:57:29.829 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:29.886 INFO:tasks.cephadm:Adding host smithi179 to orchestrator... 2024-03-08T23:57:29.887 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch host add smithi179 2024-03-08T23:57:30.125 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:30.499 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:30 smithi073 ceph-mon[32309]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:57:30.499 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:30.499 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:30 smithi073 ceph-mon[32309]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- 172.21.15.73:0/1637332399 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 msgr2=0x7f2698105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 --2- 172.21.15.73:0/1637332399 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f2698105a70 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2688009910 tx=0x7f2688009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- 172.21.15.73:0/1637332399 shutdown_connections 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 --2- 172.21.15.73:0/1637332399 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f2698105a70 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- 172.21.15.73:0/1637332399 >> 172.21.15.73:0/1637332399 conn(0x7f26980fb930 msgr2=0x7f26980fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- 172.21.15.73:0/1637332399 shutdown_connections 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- 172.21.15.73:0/1637332399 wait complete. 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 Processor -- start 2024-03-08T23:57:31.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.723+0000 7f269e36d700 1 -- start start 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f269e36d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f269e36d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f269810ad30 con 0x7f2698105690 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f2697fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f2697fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46058/0 (socket says 172.21.15.73:46058) 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f2697fff700 1 -- 172.21.15.73:0/2857674697 learned_addr learned my addr 172.21.15.73:0/2857674697 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:31.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f2697fff700 1 -- 172.21.15.73:0/2857674697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26981966f0 con 0x7f2698105690 2024-03-08T23:57:31.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f2697fff700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2688005850 tx=0x7f2688005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:31.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2688028070 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.724+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f268800bc30 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.725+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2698196980 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.725+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2688003a20 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.725+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f269819aa80 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.725+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f268802b070 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.725+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2698101e70 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.726+0000 7f26957fa700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 0x7f268003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.726+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2688020070 con 0x7f2698105690 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.726+0000 7f26977fe700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 0x7f268003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:31.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.726+0000 7f26977fe700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 0x7f268003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f268c006fd0 tx=0x7f268c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:31.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.731+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2688013070 con 0x7f2698105690 2024-03-08T23:57:31.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:31.932+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}) v1 -- 0x7f2698063410 con 0x7f2680038fc0 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:33.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:33.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:57:33.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:33.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:33 smithi073 ceph-mon[32309]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-08T23:57:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:33 smithi073 ceph-mon[32309]: Deploying cephadm binary to smithi179 2024-03-08T23:57:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:33 smithi073 ceph-mon[32309]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:33 smithi073 ceph-mon[32309]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:35.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:34 smithi073 ceph-mon[32309]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:57:35.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:34 smithi073 ceph-mon[32309]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:35.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.978+0000 7f26957fa700 1 -- 172.21.15.73:0/2857674697 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f2698063410 con 0x7f2680038fc0 2024-03-08T23:57:35.974 INFO:teuthology.orchestra.run.smithi073.stdout:Added host 'smithi179' with addr '172.21.15.179' 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 msgr2=0x7f268003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 0x7f268003b480 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f268c006fd0 tx=0x7f268c006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 msgr2=0x7f26981961b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f2688005850 tx=0x7f2688005030 comp rx=0 tx=0).stop 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 shutdown_connections 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2680038fc0 0x7f268003b480 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:35.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 --2- 172.21.15.73:0/2857674697 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2698105690 0x7f26981961b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:35.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 >> 172.21.15.73:0/2857674697 conn(0x7f26980fb930 msgr2=0x7f2698108140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:35.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 shutdown_connections 2024-03-08T23:57:35.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:35.980+0000 7f269e36d700 1 -- 172.21.15.73:0/2857674697 wait complete. 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: Added host smithi179 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:36.546 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:36.574 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch host ls --format=json 2024-03-08T23:57:36.808 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 -- 172.21.15.73:0/2148826491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 msgr2=0x7fee201003e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 --2- 172.21.15.73:0/2148826491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201003e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fee10009910 tx=0x7fee10009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 -- 172.21.15.73:0/2148826491 shutdown_connections 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 --2- 172.21.15.73:0/2148826491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201003e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 -- 172.21.15.73:0/2148826491 >> 172.21.15.73:0/2148826491 conn(0x7fee20076d90 msgr2=0x7fee200771a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 -- 172.21.15.73:0/2148826491 shutdown_connections 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.158+0000 7fee25de4700 1 -- 172.21.15.73:0/2148826491 wait complete. 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee25de4700 1 Processor -- start 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee25de4700 1 -- start start 2024-03-08T23:57:38.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee25de4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee25de4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee2010be90 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee1f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee1f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46076/0 (socket says 172.21.15.73:46076) 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.159+0000 7fee1f7fe700 1 -- 172.21.15.73:0/3354513398 learned_addr learned my addr 172.21.15.73:0/3354513398 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee1f7fe700 1 -- 172.21.15.73:0/3354513398 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20197830 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee1f7fe700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee10005850 tx=0x7fee10005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee10026070 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee20197b20 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee1000bbd0 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fee10003970 con 0x7fee20100000 2024-03-08T23:57:38.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.160+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee2019bbc0 con 0x7fee20100000 2024-03-08T23:57:38.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.161+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fee10029070 con 0x7fee20100000 2024-03-08T23:57:38.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.161+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee201030c0 con 0x7fee20100000 2024-03-08T23:57:38.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.161+0000 7fee1cff9700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 0x7fee0803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:38.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.161+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fee1001e070 con 0x7fee20100000 2024-03-08T23:57:38.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.161+0000 7fee1effd700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 0x7fee0803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:38.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.162+0000 7fee1effd700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 0x7fee0803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee14006fd0 tx=0x7fee14006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:38.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.166+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fee10013070 con 0x7fee20100000 2024-03-08T23:57:38.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.367+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fee20198650 con 0x7fee08038fc0 2024-03-08T23:57:38.364 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:38.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.369+0000 7fee1cff9700 1 -- 172.21.15.73:0/3354513398 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+240 (secure 0 0 0) 0x7fee20198650 con 0x7fee08038fc0 2024-03-08T23:57:38.365 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-08T23:57:38.365 INFO:teuthology.orchestra.run.smithi073.stdout:[{"addr": "172.21.15.73", "hostname": "smithi073", "labels": [], "status": ""}, {"addr": "172.21.15.102", "hostname": "smithi102", "labels": [], "status": ""}, {"addr": "172.21.15.179", "hostname": "smithi179", "labels": [], "status": ""}] 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 msgr2=0x7fee0803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 0x7fee0803b480 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fee14006fd0 tx=0x7fee14006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 msgr2=0x7fee201972f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fee10005850 tx=0x7fee10005030 comp rx=0 tx=0).stop 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 shutdown_connections 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fee08038fc0 0x7fee0803b480 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 --2- 172.21.15.73:0/3354513398 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee20100000 0x7fee201972f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 >> 172.21.15.73:0/3354513398 conn(0x7fee20076d90 msgr2=0x7fee200fe020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 shutdown_connections 2024-03-08T23:57:38.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:38.372+0000 7fee25de4700 1 -- 172.21.15.73:0/3354513398 wait complete. 2024-03-08T23:57:38.942 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-08T23:57:38.942 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd crush tunables default 2024-03-08T23:57:39.179 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:57:39.300 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:39 smithi073 ceph-mon[32309]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:39.300 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:39 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:40.306 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:40 smithi073 ceph-mon[32309]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 -- 172.21.15.73:0/55065965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 msgr2=0x7f56c0107ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/55065965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0107ba0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f56bc009910 tx=0x7f56bc009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 -- 172.21.15.73:0/55065965 shutdown_connections 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/55065965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0107ba0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 -- 172.21.15.73:0/55065965 >> 172.21.15.73:0/55065965 conn(0x7f56c00fb930 msgr2=0x7f56c00fdd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 -- 172.21.15.73:0/55065965 shutdown_connections 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.714+0000 7f56c7b1f700 1 -- 172.21.15.73:0/55065965 wait complete. 2024-03-08T23:57:40.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c7b1f700 1 Processor -- start 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c7b1f700 1 -- start start 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c7b1f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c7b1f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56c010bda0 con 0x7f56c01077c0 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c58bb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c58bb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38642/0 (socket says 172.21.15.73:38642) 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c58bb700 1 -- 172.21.15.73:0/1486431257 learned_addr learned my addr 172.21.15.73:0/1486431257 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:57:40.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c58bb700 1 -- 172.21.15.73:0/1486431257 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56c0197780 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.715+0000 7f56c58bb700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006010 tx=0x7f56bc005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56bc026070 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56bc00bc30 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56c0197a70 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f56bc0039d0 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56c019bb10 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.716+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f56bc029070 con 0x7f56c01077c0 2024-03-08T23:57:40.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.717+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56c0102ff0 con 0x7f56c01077c0 2024-03-08T23:57:40.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.717+0000 7f56b6ffd700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 0x7f56ac03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:40.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.717+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56bc01e070 con 0x7f56c01077c0 2024-03-08T23:57:40.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.717+0000 7f56c50ba700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 0x7f56ac03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:40.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.717+0000 7f56c50ba700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 0x7f56ac03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:40.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.722+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f56bc013070 con 0x7f56c01077c0 2024-03-08T23:57:40.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:40.921+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f56c0063410 con 0x7f56c01077c0 2024-03-08T23:57:41.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.033+0000 7f56b6ffd700 1 -- 172.21.15.73:0/1486431257 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f56bc030090 con 0x7f56c01077c0 2024-03-08T23:57:41.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 msgr2=0x7f56ac03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 0x7f56ac03b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f56b0006fd0 tx=0x7f56b0006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 msgr2=0x7f56c0197240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f56bc006010 tx=0x7f56bc005030 comp rx=0 tx=0).stop 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 shutdown_connections 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ac038fc0 0x7f56ac03b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 --2- 172.21.15.73:0/1486431257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f56c01077c0 0x7f56c0197240 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 >> 172.21.15.73:0/1486431257 conn(0x7f56c00fb930 msgr2=0x7f56c00fbd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 shutdown_connections 2024-03-08T23:57:41.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:57:41.035+0000 7f56c7b1f700 1 -- 172.21.15.73:0/1486431257 wait complete. 2024-03-08T23:57:41.032 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2024-03-08T23:57:41.353 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:41 smithi073 ceph-mon[32309]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:41.353 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:41 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1486431257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-08T23:57:41.680 INFO:tasks.cephadm:Adding mon.a on smithi073 2024-03-08T23:57:41.680 INFO:tasks.cephadm:Adding mon.b on smithi102 2024-03-08T23:57:41.680 INFO:tasks.cephadm:Adding mon.c on smithi179 2024-03-08T23:57:41.680 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch apply mon '3;smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c' 2024-03-08T23:57:41.913 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:42.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1486431257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:42.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:42 smithi073 ceph-mon[32309]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:57:43.382 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 -- 172.21.15.179:0/1393630115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 msgr2=0x7f074c106aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 --2- 172.21.15.179:0/1393630115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c106aa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f0748009910 tx=0x7f0748009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 -- 172.21.15.179:0/1393630115 shutdown_connections 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 --2- 172.21.15.179:0/1393630115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c106aa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 -- 172.21.15.179:0/1393630115 >> 172.21.15.179:0/1393630115 conn(0x7f074c0fc1d0 msgr2=0x7f074c0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 -- 172.21.15.179:0/1393630115 shutdown_connections 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.381+0000 7f0754287700 1 -- 172.21.15.179:0/1393630115 wait complete. 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0754287700 1 Processor -- start 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0754287700 1 -- start start 2024-03-08T23:57:43.383 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0754287700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0754287700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f074c10ad00 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0752023700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0752023700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.179:41100/0 (socket says 172.21.15.179:41100) 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0752023700 1 -- 172.21.15.179:0/3241031699 learned_addr learned my addr 172.21.15.179:0/3241031699 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0752023700 1 -- 172.21.15.179:0/3241031699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f074c1977d0 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.382+0000 7f0752023700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0748005850 tx=0x7f0748005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0748026070 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f074c197ac0 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f074800bbd0 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0748003970 con 0x7f074c1066c0 2024-03-08T23:57:43.384 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f074c19bb60 con 0x7f074c1066c0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.383+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f074c101e00 con 0x7f074c1066c0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.384+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f0748029070 con 0x7f074c1066c0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.384+0000 7f07437fe700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 0x7f073803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.384+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f074801e070 con 0x7f074c1066c0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.384+0000 7f0751822700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 0x7f073803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.384+0000 7f0751822700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 0x7f073803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f073c006fd0 tx=0x7f073c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:43.385 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.389+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0748013070 con 0x7f074c1066c0 2024-03-08T23:57:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:43.586 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.591+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f074c1985f0 con 0x7f0738038fc0 2024-03-08T23:57:43.593 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.597+0000 7f07437fe700 1 -- 172.21.15.179:0/3241031699 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f074c1985f0 con 0x7f0738038fc0 2024-03-08T23:57:43.593 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mon update... 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 msgr2=0x7f073803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 0x7f073803b480 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f073c006fd0 tx=0x7f073c006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 msgr2=0x7f074c197290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f0748005850 tx=0x7f0748005030 comp rx=0 tx=0).stop 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 shutdown_connections 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0738038fc0 0x7f073803b480 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 --2- 172.21.15.179:0/3241031699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f074c1066c0 0x7f074c197290 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:43.595 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 >> 172.21.15.179:0/3241031699 conn(0x7f074c0fc1d0 msgr2=0x7f074c108200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:43.596 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 shutdown_connections 2024-03-08T23:57:43.596 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:43.599+0000 7f0754287700 1 -- 172.21.15.179:0/3241031699 wait complete. 2024-03-08T23:57:44.196 DEBUG:teuthology.orchestra.run.smithi102:mon.b> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.b.service 2024-03-08T23:57:44.199 DEBUG:teuthology.orchestra.run.smithi179:mon.c> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.c.service 2024-03-08T23:57:44.201 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T23:57:44.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T23:57:44.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:44.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: Saving service mon spec with placement smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c;count:3 2024-03-08T23:57:44.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:44.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: Deploying daemon mon.c on smithi179 2024-03-08T23:57:44.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:44 smithi073 ceph-mon[32309]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:45.354 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:46 smithi073 ceph-mon[32309]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:48.105 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- 172.21.15.179:0/3090230473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 msgr2=0x7fdd70102ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 --2- 172.21.15.179:0/3090230473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70102ce0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009910 tx=0x7fdd60009c20 comp rx=0 tx=0).stop 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- 172.21.15.179:0/3090230473 shutdown_connections 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 --2- 172.21.15.179:0/3090230473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70102ce0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- 172.21.15.179:0/3090230473 >> 172.21.15.179:0/3090230473 conn(0x7fdd700fc1e0 msgr2=0x7fdd700fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- 172.21.15.179:0/3090230473 shutdown_connections 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- 172.21.15.179:0/3090230473 wait complete. 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 Processor -- start 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.107+0000 7fdd767e1700 1 -- start start 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd767e1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:48.508 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd767e1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd7010be60 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd6ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd6ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.179:54282/0 (socket says 172.21.15.179:54282) 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd6ffff700 1 -- 172.21.15.179:0/1496504655 learned_addr learned my addr 172.21.15.179:0/1496504655 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.108+0000 7fdd6ffff700 1 -- 172.21.15.179:0/1496504655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd70197880 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd6ffff700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd60005850 tx=0x7fdd60005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60026070 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd6000bc30 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd70197b10 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fdd60003a20 con 0x7fdd70102900 2024-03-08T23:57:48.509 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.109+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd70197f70 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.110+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fdd60029070 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.110+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd7006a4a0 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.110+0000 7fdd6d7fa700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 0x7fdd5803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.110+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd6001e070 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.110+0000 7fdd6f7fe700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 0x7fdd5803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.111+0000 7fdd6f7fe700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 0x7fdd5803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd64006fd0 tx=0x7fdd64006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.115+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdd60013070 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.429+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd70198650 con 0x7fdd70102900 2024-03-08T23:57:48.511 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.429+0000 7fdd6d7fa700 1 -- 172.21.15.179:0/1496504655 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+745 (secure 0 0 0) 0x7fdd60030020 con 0x7fdd70102900 2024-03-08T23:57:48.512 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:57:48.512 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":1,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","modified":"2024-03-08T23:56:05.299609Z","created":"2024-03-08T23:56:05.299609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.431+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 msgr2=0x7fdd5803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.431+0000 7fdd767e1700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 0x7fdd5803b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fdd64006fd0 tx=0x7fdd64006e40 comp rx=0 tx=0).stop 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.431+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 msgr2=0x7fdd70197340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.431+0000 7fdd767e1700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7fdd60005850 tx=0x7fdd60005030 comp rx=0 tx=0).stop 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 shutdown_connections 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd58038fc0 0x7fdd5803b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 --2- 172.21.15.179:0/1496504655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd70102900 0x7fdd70197340 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 >> 172.21.15.179:0/1496504655 conn(0x7fdd700fc1e0 msgr2=0x7fdd70103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 shutdown_connections 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:48.432+0000 7fdd767e1700 1 -- 172.21.15.179:0/1496504655 wait complete. 2024-03-08T23:57:48.513 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 1 2024-03-08T23:57:49.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:48 smithi179 systemd[1]: Started Ceph mon.c for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:57:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:48 smithi073 ceph-mon[32309]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:48 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/1496504655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:57:50.250 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T23:57:50.250 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T23:57:50.486 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: mon.a calling monitor election 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: mon.c calling monitor election 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: monmap e2: 2 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: fsmap 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: mgrmap e13: x(active, since 50s) 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: overall HEALTH_OK 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:54.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:57:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: mon.a calling monitor election 2024-03-08T23:57:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: mon.c calling monitor election 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: monmap e2: 2 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: fsmap 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: mgrmap e13: x(active, since 50s) 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: overall HEALTH_OK 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 -- 172.21.15.179:0/3548228720 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 msgr2=0x7f319c005610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3548228720 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f319c005610 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009bf0 tx=0x7f31a8009f00 comp rx=0 tx=0).stop 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 -- 172.21.15.179:0/3548228720 shutdown_connections 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3548228720 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f319c005610 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 -- 172.21.15.179:0/3548228720 >> 172.21.15.179:0/3548228720 conn(0x7f31b00fc2f0 msgr2=0x7f31b00fe710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 -- 172.21.15.179:0/3548228720 shutdown_connections 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.894+0000 7f31b85fb700 1 -- 172.21.15.179:0/3548228720 wait complete. 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.895+0000 7f31b85fb700 1 Processor -- start 2024-03-08T23:57:54.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.895+0000 7f31b85fb700 1 -- start start 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b85fb700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f31b0197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b85fb700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b85fb700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b0109970 con 0x7f31b0102b00 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b85fb700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b01097f0 con 0x7f31b0197a40 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:54.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:46768/0 (socket says 172.21.15.179:46768) 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 -- 172.21.15.179:0/3433352634 learned_addr learned my addr 172.21.15.179:0/3433352634 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b6397700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f31b0197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 -- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 msgr2=0x7f31b019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 -- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 msgr2=0x7f31b019bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.896+0000 7f31b5b96700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b6397700 1 -- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 msgr2=0x7f31b019bd30 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b6397700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b6397700 1 -- 172.21.15.179:0/3433352634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b0198080 con 0x7f31b0102b00 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b6397700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f31b0197500 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009bf0 tx=0x7f31a8003e00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f31a8018c30 con 0x7f31b0102b00 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31a8018dd0 con 0x7f31b0102b00 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b019c270 con 0x7f31b0102b00 2024-03-08T23:57:54.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f31a8020ad0 con 0x7f31b0102b00 2024-03-08T23:57:54.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.897+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b019c7d0 con 0x7f31b0102b00 2024-03-08T23:57:54.896 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.898+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f31a801e070 con 0x7f31b0102b00 2024-03-08T23:57:54.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.899+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b006a840 con 0x7f31b0102b00 2024-03-08T23:57:54.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.899+0000 7f31a77fe700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 0x7f319403b270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:57:54.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.899+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f31a801c070 con 0x7f31b0102b00 2024-03-08T23:57:54.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.899+0000 7f31b5b96700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 0x7f319403b270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:57:54.897 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.900+0000 7f31b5b96700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 0x7f319403b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009180 tx=0x7f31a0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:57:54.901 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:54.904+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f31a8030bf0 con 0x7f31b0102b00 2024-03-08T23:57:55.183 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.186+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31b019ca80 con 0x7f31b0102b00 2024-03-08T23:57:55.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:55.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:55.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:57:55.184 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:55.184 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: Deploying daemon mon.b on smithi102 2024-03-08T23:57:55.184 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:55.184 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.187+0000 7f31a77fe700 1 -- 172.21.15.179:0/3433352634 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1015 (secure 0 0 0) 0x7f31a801e360 con 0x7f31b0102b00 2024-03-08T23:57:55.184 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:57:55.184 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":2,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","modified":"2024-03-08T23:57:48.922807Z","created":"2024-03-08T23:56:05.299609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-08T23:57:55.185 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 msgr2=0x7f319403b270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 0x7f319403b270 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f31a0009180 tx=0x7f31a0008040 comp rx=0 tx=0).stop 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 msgr2=0x7f31b0197500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f31b0197500 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009bf0 tx=0x7f31a8003e00 comp rx=0 tx=0).stop 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 shutdown_connections 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31b0102b00 0x7f31b0197500 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3194038db0 0x7f319403b270 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 --2- 172.21.15.179:0/3433352634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31b0197a40 0x7f31b019bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 >> 172.21.15.179:0/3433352634 conn(0x7f31b00fc2f0 msgr2=0x7f31b00fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 shutdown_connections 2024-03-08T23:57:55.186 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:57:55.189+0000 7f31b85fb700 1 -- 172.21.15.179:0/3433352634 wait complete. 2024-03-08T23:57:55.187 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 2 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: Deploying daemon mon.b on smithi102 2024-03-08T23:57:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:56 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/3433352634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:57:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:56 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/3433352634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:57:56.752 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-08T23:57:56.753 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph mon dump -f json 2024-03-08T23:57:56.996 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:57:57.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:57 smithi179 ceph-mon[36637]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:57.303 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:57 smithi073 ceph-mon[32309]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.046 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 podman[36086]: 2024-03-08 23:57:58.027952343 +0000 UTC m=+0.351503535 container start feeab1859012fcf247f8987bf9984d4d816ae53631946e0ad1eeebfb1997fe6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-08T23:57:58.293 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:57:58 smithi179 ceph-mon[36637]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:57:58 smithi073 ceph-mon[32309]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: set uid:gid to 167:167 (ceph:ceph) 2024-03-08T23:57:58.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev), process ceph-mon, pid 2 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pidfile_write: ignore empty --pid-file 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: load: jerasure load: lrc 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: RocksDB version: 7.9.2 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Git sha 0 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Compile date 2024-03-08 15:06:51 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: DB SUMMARY 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: DB Session ID: RG4O2T816WYIJU96GJRW 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: CURRENT file: CURRENT 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: IDENTITY file: IDENTITY 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-08T23:57:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.error_if_exists: 0 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.create_if_missing: 0 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.paranoid_checks: 1 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.env: 0x5593b2230b20 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.fs: PosixFileSystem 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.info_log: 0x5593b49cd8e0 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_file_opening_threads: 16 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.statistics: (nil) 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.use_fsync: 0 2024-03-08T23:57:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_log_file_size: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.keep_log_file_num: 1000 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.recycle_log_file_num: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_fallocate: 1 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_mmap_reads: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_mmap_writes: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.use_direct_reads: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.create_missing_column_families: 0 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.db_log_dir: 2024-03-08T23:57:58.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.wal_dir: 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.advise_random_on_open: 1 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.db_write_buffer_size: 0 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.write_buffer_manager: 0x5593b49a1400 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-08T23:57:58.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.rate_limiter: (nil) 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.wal_recovery_mode: 2 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enable_thread_tracking: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enable_pipelined_write: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.unordered_write: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.row_cache: None 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.wal_filter: None 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_ingest_behind: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.two_write_queues: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.manual_wal_flush: 0 2024-03-08T23:57:58.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.wal_compression: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.atomic_flush: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.log_readahead_size: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.best_efforts_recovery: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.allow_data_in_errors: 0 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.db_host_id: __hostname__ 2024-03-08T23:57:58.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_background_jobs: 2 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_background_compactions: -1 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_subcompactions: 1 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_total_wal_size: 0 2024-03-08T23:57:58.518 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_open_files: -1 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bytes_per_sync: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_readahead_size: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_background_flushes: -1 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Compression algorithms supported: 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kZSTD supported: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kXpressCompression supported: 0 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kLZ4HCCompression supported: 1 2024-03-08T23:57:58.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kZlibCompression supported: 1 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kSnappyCompression supported: 1 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kLZ4Compression supported: 1 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: kBZip2Compression supported: 0 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.merge_operator: 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_filter: None 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_filter_factory: None 2024-03-08T23:57:58.520 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.sst_partitioner_factory: None 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5593b49cd9e0) 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: cache_index_and_filter_blocks: 1 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: pin_top_level_index_and_filter: 1 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: index_type: 0 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: data_block_index_type: 0 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: index_shortening: 1 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: checksum: 4 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: no_block_cache: 0 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_cache: 0x5593b49d3090 2024-03-08T23:57:58.521 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_cache_name: BinnedLRUCache 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_cache_options: 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: capacity : 536870912 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: num_shard_bits : 4 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: strict_capacity_limit : 0 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: high_pri_pool_ratio: 0.000 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_cache_compressed: (nil) 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: persistent_cache: (nil) 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_size: 4096 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_size_deviation: 10 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_restart_interval: 16 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: index_block_restart_interval: 1 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: metadata_block_size: 4096 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: partition_filters: 0 2024-03-08T23:57:58.522 INFO:journalctl@ceph.mon.b.smithi102.stdout: use_delta_encoding: 1 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: filter_policy: bloomfilter 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: whole_key_filtering: 1 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: verify_compression: 0 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: read_amp_bytes_per_bit: 0 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: format_version: 5 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: enable_index_compression: 1 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: block_align: 0 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: max_auto_readahead_size: 262144 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: prepopulate_block_cache: 0 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: initial_auto_readahead_size: 8192 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout: num_file_reads_for_auto_readahead: 2 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.write_buffer_size: 33554432 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_write_buffer_number: 2 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression: NoCompression 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression: Disabled 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.prefix_extractor: nullptr 2024-03-08T23:57:58.523 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.num_levels: 7 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.level: 32767 2024-03-08T23:57:58.524 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.strategy: 0 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.enabled: false 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.target_file_size_base: 67108864 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-08T23:57:58.525 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.arena_block_size: 1048576 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-08T23:57:58.526 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.disable_auto_compactions: 0 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.table_properties_collectors: 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.inplace_update_support: 0 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-08T23:57:58.527 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.bloom_locality: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.max_successive_merges: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.paranoid_file_checks: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.force_consistency_checks: 1 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.report_bg_io_stats: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.ttl: 2592000 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enable_blob_files: false 2024-03-08T23:57:58.528 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.min_blob_size: 0 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_file_size: 268435456 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.blob_file_starting_level: 0 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 9fc34c7a-447a-442b-b012-507b7bea0cc9 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942278057127, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-08T23:57:58.529 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942278057529, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1709942278, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "9fc34c7a-447a-442b-b012-507b7bea0cc9", "db_session_id": "RG4O2T816WYIJU96GJRW", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: EVENT_LOG_v1 {"time_micros": 1709942278057604, "job": 1, "event": "recovery_finished"} 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5593b4ac6000 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: DB pointer 0x5593b4ab0000 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: ** DB Stats ** 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T23:57:58.530 INFO:journalctl@ceph.mon.b.smithi102.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: ** Compaction Stats [default] ** 2024-03-08T23:57:58.531 INFO:journalctl@ceph.mon.b.smithi102.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.4 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Cumulative compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Interval compaction: 0.00 GB write, 0.71 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-08T23:57:58.532 INFO:journalctl@ceph.mon.b.smithi102.stdout: Block cache BinnedLRUCache@0x5593b49d3090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 9e-06 secs_since: 0 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: using public_addr v2:172.21.15.102:0/0 -> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: starting mon.b rank -1 at public addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] at bind addrs [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(???) e0 preinit fsid 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).mds e1 new map 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: e1 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-08T23:57:58.533 INFO:journalctl@ceph.mon.b.smithi102.stdout: legacy client fscid: -1 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout: 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout: No filesystems configured 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Added host smithi102 2024-03-08T23:57:58.534 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi179", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi102", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:57:58.535 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Deploying cephadm binary to smithi179 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.536 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Added host smithi179 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1486431257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1486431257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.537 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi179", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Saving service mon spec with placement smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c;count:3 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.538 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Deploying daemon mon.c on smithi179 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/1496504655' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.a calling monitor election 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.539 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.c calling monitor election 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: monmap e2: 2 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: fsmap 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mgrmap e13: x(active, since 50s) 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: overall HEALTH_OK 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.540 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: Deploying daemon mon.b on smithi102 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/3433352634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 ceph-mon[36106]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 bash[36086]: feeab1859012fcf247f8987bf9984d4d816ae53631946e0ad1eeebfb1997fe6b 2024-03-08T23:57:58.541 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:57:58 smithi102 systemd[1]: Started Ceph mon.b for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: mon.a calling monitor election 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: mon.c calling monitor election 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: mon.b calling monitor election 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: monmap e3: 3 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],b=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: fsmap 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: mgrmap e13: x(active, since 59s) 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: overall HEALTH_OK 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:03 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: mon.a calling monitor election 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: mon.c calling monitor election 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: mon.b calling monitor election 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: monmap e3: 3 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],b=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: fsmap 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: mgrmap e13: x(active, since 59s) 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: overall HEALTH_OK 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:03 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:58:03.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: mon.a calling monitor election 2024-03-08T23:58:03.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: mon.c calling monitor election 2024-03-08T23:58:03.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: mon.b calling monitor election 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: monmap e3: 3 mons at {a=[v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0],b=[v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0],c=[v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: fsmap 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: osdmap e4: 0 total, 0 up, 0 in 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: mgrmap e13: x(active, since 59s) 2024-03-08T23:58:03.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: overall HEALTH_OK 2024-03-08T23:58:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:03 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:04.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:04.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:04 smithi179 ceph-mon[36637]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:04.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:04.493 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 -- 172.21.15.179:0/2836870350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 msgr2=0x7fdd7800a920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 --2- 172.21.15.179:0/2836870350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd7800a920 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fdd8c101880 tx=0x7fdd8400fe90 comp rx=0 tx=0).stop 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 -- 172.21.15.179:0/2836870350 shutdown_connections 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 --2- 172.21.15.179:0/2836870350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd7800a920 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 -- 172.21.15.179:0/2836870350 >> 172.21.15.179:0/2836870350 conn(0x7fdd8c0fc2d0 msgr2=0x7fdd8c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 -- 172.21.15.179:0/2836870350 shutdown_connections 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.496+0000 7fdd92e30700 1 -- 172.21.15.179:0/2836870350 wait complete. 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 Processor -- start 2024-03-08T23:58:04.496 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 -- start start 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd8c19f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd8c19fae0 0x7fdd8c199670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c1077e0 con 0x7fdd780084b0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c107660 con 0x7fdd8c19fae0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.497+0000 7fdd92e30700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd8c107960 con 0x7fdd8c199bb0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:48718/0 (socket says 172.21.15.179:48718) 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 -- 172.21.15.179:0/3018404196 learned_addr learned my addr 172.21.15.179:0/3018404196 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd8bfff700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd8c19fae0 0x7fdd8c199670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd90bcc700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd8c19f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 -- 172.21.15.179:0/3018404196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd8c19fae0 msgr2=0x7fdd8c199670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd8c19fae0 0x7fdd8c199670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.497 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 -- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 msgr2=0x7fdd8c19f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd8c19f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 -- 172.21.15.179:0/3018404196 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd8c1a6350 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd913cd700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400b550 tx=0x7fdd7400b860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74015020 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd74011d10 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c1a65e0 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.498+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd74010dd0 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.499+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd8c1a6b20 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.499+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fdd74018020 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.499+0000 7fdd89ffb700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 0x7fdd6c03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.499+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd7404c880 con 0x7fdd8c199bb0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.500+0000 7fdd90bcc700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 0x7fdd6c03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:04.498 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.500+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd8c101f00 con 0x7fdd8c199bb0 2024-03-08T23:58:04.499 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.500+0000 7fdd90bcc700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 0x7fdd6c03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd84015040 tx=0x7fdd84001e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:04.501 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.505+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdd7401f020 con 0x7fdd8c199bb0 2024-03-08T23:58:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:04 smithi102 ceph-mon[36106]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:04.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:04.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:04 smithi073 ceph-mon[32309]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:04.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:04.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:58:04.793 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.796+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd8c1a6de0 con 0x7fdd8c199bb0 2024-03-08T23:58:04.793 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.797+0000 7fdd89ffb700 1 -- 172.21.15.179:0/3018404196 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1285 (secure 0 0 0) 0x7fdd74018310 con 0x7fdd8c199bb0 2024-03-08T23:58:04.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2024-03-08T23:58:04.794 INFO:teuthology.orchestra.run.smithi179.stdout:{"epoch":3,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","modified":"2024-03-08T23:57:58.079052Z","created":"2024-03-08T23:56:05.299609Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:3300","nonce":0},{"type":"v1","addr":"172.21.15.73:6789","nonce":0}]},"addr":"172.21.15.73:6789/0","public_addr":"172.21.15.73:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:3300","nonce":0},{"type":"v1","addr":"172.21.15.179:6789","nonce":0}]},"addr":"172.21.15.179:6789/0","public_addr":"172.21.15.179:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:3300","nonce":0},{"type":"v1","addr":"172.21.15.102:6789","nonce":0}]},"addr":"172.21.15.102:6789/0","public_addr":"172.21.15.102:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1,2]} 2024-03-08T23:58:04.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 msgr2=0x7fdd6c03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:04.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 0x7fdd6c03b910 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fdd84015040 tx=0x7fdd84001e70 comp rx=0 tx=0).stop 2024-03-08T23:58:04.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 msgr2=0x7fdd8c19a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:04.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd7400b550 tx=0x7fdd7400b860 comp rx=0 tx=0).stop 2024-03-08T23:58:04.796 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 shutdown_connections 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd780084b0 0x7fdd8c19f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdd6c039450 0x7fdd6c03b910 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd8c199bb0 0x7fdd8c19a060 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 --2- 172.21.15.179:0/3018404196 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd8c19fae0 0x7fdd8c199670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 >> 172.21.15.179:0/3018404196 conn(0x7fdd8c0fc2d0 msgr2=0x7fdd8c1022c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 shutdown_connections 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:04.800+0000 7fdd92e30700 1 -- 172.21.15.179:0/3018404196 wait complete. 2024-03-08T23:58:04.797 INFO:teuthology.orchestra.run.smithi179.stderr:dumped monmap epoch 3 2024-03-08T23:58:05.543 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-08T23:58:05.544 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph config generate-minimal-conf 2024-03-08T23:58:05.792 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:58:05.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/3018404196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:05.834 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/3018404196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:58:05.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:05.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-08T23:58:06.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/3018404196' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.012 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:06.012 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:06.012 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:06.012 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:06.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:06 smithi179 ceph-mon[36637]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T23:58:06.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:06 smithi179 ceph-mon[36637]: Reconfiguring daemon mon.a on smithi073 2024-03-08T23:58:06.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:06 smithi179 ceph-mon[36637]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:06.843 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:06 smithi073 ceph-mon[32309]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T23:58:06.843 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:06 smithi073 ceph-mon[32309]: Reconfiguring daemon mon.a on smithi073 2024-03-08T23:58:06.843 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:06 smithi073 ceph-mon[32309]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:07.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:06 smithi102 ceph-mon[36106]: Reconfiguring mon.a (unknown last config time)... 2024-03-08T23:58:07.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:06 smithi102 ceph-mon[36106]: Reconfiguring daemon mon.a on smithi073 2024-03-08T23:58:07.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:06 smithi102 ceph-mon[36106]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 -- 172.21.15.73:0/3139464060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 msgr2=0x7fdf38106a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/3139464060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf38106a70 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fdf280099c0 tx=0x7fdf28009cd0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 -- 172.21.15.73:0/3139464060 shutdown_connections 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/3139464060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 0x7fdf3810ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/3139464060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 0x7fdf38100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/3139464060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf38106a70 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.371+0000 7fdf3fc23700 1 -- 172.21.15.73:0/3139464060 >> 172.21.15.73:0/3139464060 conn(0x7fdf38076040 msgr2=0x7fdf38076450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- 172.21.15.73:0/3139464060 shutdown_connections 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- 172.21.15.73:0/3139464060 wait complete. 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 Processor -- start 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- start start 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 0x7fdf381a0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 0x7fdf3819a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf381120f0 con 0x7fdf38101330 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38111f70 con 0x7fdf381071a0 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.372+0000 7fdf3fc23700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf38112270 con 0x7fdf38100980 2024-03-08T23:58:08.371 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51120/0 (socket says 172.21.15.73:51120) 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 -- 172.21.15.73:0/664025925 learned_addr learned my addr 172.21.15.73:0/664025925 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3e1c0700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 0x7fdf3819a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d9bf700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 0x7fdf381a0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 -- 172.21.15.73:0/664025925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 msgr2=0x7fdf381a0410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 0x7fdf381a0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 -- 172.21.15.73:0/664025925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 msgr2=0x7fdf3819a5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 0x7fdf3819a5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 -- 172.21.15.73:0/664025925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf3819ad30 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3d1be700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800b580 tx=0x7fdf280038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28027070 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdf28003e40 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf3819afc0 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdf28022d10 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.373+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf381a72e0 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.374+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7fdf2802a070 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.374+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf38068aa0 con 0x7fdf38101330 2024-03-08T23:58:08.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.374+0000 7fdf2effd700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 0x7fdf2403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:08.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.374+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdf2804d060 con 0x7fdf38101330 2024-03-08T23:58:08.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.374+0000 7fdf3d9bf700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 0x7fdf2403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:08.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.375+0000 7fdf3d9bf700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 0x7fdf2403b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009550 tx=0x7fdf30009040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:08.375 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.379+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdf28013070 con 0x7fdf38101330 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.583+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fdf3802d830 con 0x7fdf38101330 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.583+0000 7fdf2effd700 1 -- 172.21.15.73:0/664025925 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+275 (secure 0 0 0) 0x7fdf28031090 con 0x7fdf38101330 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stdout:# minimal ceph.conf for 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-08T23:58:08.579 INFO:teuthology.orchestra.run.smithi073.stdout: mon_host = [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 msgr2=0x7fdf2403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 0x7fdf2403b6c0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fdf30009550 tx=0x7fdf30009040 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 msgr2=0x7fdf381a0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdf2800b580 tx=0x7fdf280038b0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 shutdown_connections 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdf24039200 0x7fdf2403b6c0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdf381071a0 0x7fdf3819a5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdf38100980 0x7fdf381a0410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 --2- 172.21.15.73:0/664025925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdf38101330 0x7fdf381a0950 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:08.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 >> 172.21.15.73:0/664025925 conn(0x7fdf38076040 msgr2=0x7fdf380fee40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:08.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 shutdown_connections 2024-03-08T23:58:08.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:08.585+0000 7fdf3fc23700 1 -- 172.21.15.73:0/664025925 wait complete. 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: Reconfiguring mon.b (monmap changed)... 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: Reconfiguring daemon mon.b on smithi102 2024-03-08T23:58:08.640 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:08 smithi102 ceph-mon[36106]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:08.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: Reconfiguring mon.b (monmap changed)... 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: Reconfiguring daemon mon.b on smithi102 2024-03-08T23:58:08.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:08 smithi179 ceph-mon[36637]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:08.898 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.898 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:08.898 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: Reconfiguring mon.b (monmap changed)... 2024-03-08T23:58:08.898 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:08.898 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:08.899 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:08.899 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: Reconfiguring daemon mon.b on smithi102 2024-03-08T23:58:08.899 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:08 smithi073 ceph-mon[32309]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:09.172 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-08T23:58:09.172 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:58:09.172 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T23:58:09.202 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:58:09.202 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:58:09.270 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:58:09.270 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T23:58:09.300 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:58:09.300 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:58:09.367 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:58:09.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.conf 2024-03-08T23:58:09.399 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:58:09.399 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-08T23:58:09.469 INFO:tasks.cephadm:Adding mgr.x on smithi073 2024-03-08T23:58:09.469 INFO:tasks.cephadm:Adding mgr.y on smithi102 2024-03-08T23:58:09.469 INFO:tasks.cephadm:Adding mgr.z on smithi179 2024-03-08T23:58:09.469 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch apply mgr '3;smithi073=x;smithi102=y;smithi179=z' 2024-03-08T23:58:09.743 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:58:09.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/664025925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:09.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:09.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:09.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:09.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:09.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/664025925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:10.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/664025925' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-08T23:58:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-08T23:58:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:10.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:10 smithi179 ceph-mon[36637]: Reconfiguring mon.c (monmap changed)... 2024-03-08T23:58:10.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:10 smithi179 ceph-mon[36637]: Reconfiguring daemon mon.c on smithi179 2024-03-08T23:58:10.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:10 smithi179 ceph-mon[36637]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:10.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:10 smithi073 ceph-mon[32309]: Reconfiguring mon.c (monmap changed)... 2024-03-08T23:58:10.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:10 smithi073 ceph-mon[32309]: Reconfiguring daemon mon.c on smithi179 2024-03-08T23:58:10.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:10 smithi073 ceph-mon[32309]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:10 smithi102 ceph-mon[36106]: Reconfiguring mon.c (monmap changed)... 2024-03-08T23:58:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:10 smithi102 ceph-mon[36106]: Reconfiguring daemon mon.c on smithi179 2024-03-08T23:58:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:10 smithi102 ceph-mon[36106]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 -- 172.21.15.179:0/211238334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 msgr2=0x7f28541033e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 --2- 172.21.15.179:0/211238334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f28541033e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2848009990 tx=0x7f2848009ca0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 -- 172.21.15.179:0/211238334 shutdown_connections 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 --2- 172.21.15.179:0/211238334 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285410a240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 --2- 172.21.15.179:0/211238334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 0x7f2854102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 --2- 172.21.15.179:0/211238334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f28541033e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 -- 172.21.15.179:0/211238334 >> 172.21.15.179:0/211238334 conn(0x7f28540fd3c0 msgr2=0x7f28540ff7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 -- 172.21.15.179:0/211238334 shutdown_connections 2024-03-08T23:58:12.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.344+0000 7f28590dc700 1 -- 172.21.15.179:0/211238334 wait complete. 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 Processor -- start 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 -- start start 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 0x7f2854197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285419f4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285410d970 con 0x7f2854103920 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285410d7f0 con 0x7f2854101e00 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f28590dc700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f285410daf0 con 0x7f2854102f60 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f285259c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f285259c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:43156/0 (socket says 172.21.15.179:43156) 2024-03-08T23:58:12.344 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f2852d9d700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 0x7f2854197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.345+0000 7f285259c700 1 -- 172.21.15.179:0/1811483057 learned_addr learned my addr 172.21.15.179:0/1811483057 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285359e700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285419f4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 -- 172.21.15.179:0/1811483057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 msgr2=0x7f2854197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 0x7f2854197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 -- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 msgr2=0x7f285419f4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285419f4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 -- 172.21.15.179:0/1811483057 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f285419fb50 con 0x7f2854102f60 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285359e700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f285259c700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28480059d0 tx=0x7f28480057c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2848027070 con 0x7f2854102f60 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28480041d0 con 0x7f2854102f60 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f285419fde0 con 0x7f2854102f60 2024-03-08T23:58:12.345 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f284800f6c0 con 0x7f2854102f60 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.346+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28541a0230 con 0x7f2854102f60 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.347+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f284802a070 con 0x7f2854102f60 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.347+0000 7f283bfff700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 0x7f283c03b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.347+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2854068aa0 con 0x7f2854102f60 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.347+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2848022470 con 0x7f2854102f60 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.347+0000 7f2852d9d700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 0x7f283c03b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:12.346 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.348+0000 7f2852d9d700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 0x7f283c03b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f284400ac50 tx=0x7f284400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:12.348 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.352+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2848013070 con 0x7f2854102f60 2024-03-08T23:58:12.557 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.561+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi073=x;smithi102=y;smithi179=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f2854062220 con 0x7f283c039200 2024-03-08T23:58:12.565 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.569+0000 7f283bfff700 1 -- 172.21.15.179:0/1811483057 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2854062220 con 0x7f283c039200 2024-03-08T23:58:12.565 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mgr update... 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 msgr2=0x7f283c03b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 0x7f283c03b6c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f284400ac50 tx=0x7f284400a380 comp rx=0 tx=0).stop 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 msgr2=0x7f2854198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f28480059d0 tx=0x7f28480057c0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 shutdown_connections 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2854103920 0x7f285419f4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.567 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f283c039200 0x7f283c03b6c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2854101e00 0x7f2854197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 --2- 172.21.15.179:0/1811483057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2854102f60 0x7f2854198430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:12.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 >> 172.21.15.179:0/1811483057 conn(0x7f28540fd3c0 msgr2=0x7f2854106220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:12.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 shutdown_connections 2024-03-08T23:58:12.568 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:58:12.571+0000 7f28590dc700 1 -- 172.21.15.179:0/1811483057 wait complete. 2024-03-08T23:58:12.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:12.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:12.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:12.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:12.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:12.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:12.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:13.153 DEBUG:teuthology.orchestra.run.smithi102:mgr.y> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.y.service 2024-03-08T23:58:13.156 DEBUG:teuthology.orchestra.run.smithi179:mgr.z> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z.service 2024-03-08T23:58:13.158 INFO:tasks.cephadm:Deploying OSDs... 2024-03-08T23:58:13.158 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-08T23:58:13.158 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T23:58:13.177 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T23:58:13.177 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:13.235 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-08 23:57:16.522294541 +0000 2024-03-08T23:58:13.236 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-08 23:51:52.837155949 +0000 2024-03-08T23:58:13.236 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-08 23:51:52.837155949 +0000 2024-03-08T23:58:13.236 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-08 23:51:52.837155949 +0000 2024-03-08T23:58:13.236 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T23:58:13.305 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-08T23:58:13.306 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-08T23:58:13.306 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000229644 s, 2.2 MB/s 2024-03-08T23:58:13.307 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T23:58:13.366 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-03-08T23:58:13.422 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T23:58:13.422 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:13.422 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 685 Links: 1 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-08 23:57:16.550293874 +0000 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-08 23:51:53.189147219 +0000 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-08 23:51:53.189147219 +0000 2024-03-08T23:58:13.423 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-08 23:51:53.189147219 +0000 2024-03-08T23:58:13.423 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T23:58:13.492 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-08T23:58:13.492 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-08T23:58:13.492 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000241156 s, 2.1 MB/s 2024-03-08T23:58:13.493 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T23:58:13.551 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-03-08T23:58:13.608 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T23:58:13.608 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:13.608 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-08T23:58:13.608 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:13.608 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:13.609 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-08 23:57:16.582293112 +0000 2024-03-08T23:58:13.609 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-08 23:51:53.520139005 +0000 2024-03-08T23:58:13.609 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-08 23:51:53.520139005 +0000 2024-03-08T23:58:13.609 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-08 23:51:53.520139005 +0000 2024-03-08T23:58:13.609 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T23:58:13.621 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi073=x;smithi102=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: Saving service mgr spec with placement smithi073=x;smithi102=y;smithi179=z;count:3 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:13.622 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:13 smithi179 ceph-mon[36637]: Deploying daemon mgr.z on smithi179 2024-03-08T23:58:13.676 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-08T23:58:13.677 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-08T23:58:13.677 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000288362 s, 1.8 MB/s 2024-03-08T23:58:13.678 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T23:58:13.736 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-03-08T23:58:13.792 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T23:58:13.792 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-08 23:57:16.608292493 +0000 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-08 23:51:53.880130073 +0000 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-08 23:51:53.880130073 +0000 2024-03-08T23:58:13.793 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-08 23:51:53.880130073 +0000 2024-03-08T23:58:13.793 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T23:58:13.857 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi073=x;smithi102=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:13.857 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: Saving service mgr spec with placement smithi073=x;smithi102=y;smithi179=z;count:3 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:13.858 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:13 smithi073 ceph-mon[32309]: Deploying daemon mgr.z on smithi179 2024-03-08T23:58:13.862 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-03-08T23:58:13.863 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-03-08T23:58:13.863 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000214042 s, 2.4 MB/s 2024-03-08T23:58:13.863 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T23:58:13.919 DEBUG:teuthology.orchestra.run.smithi102:> set -ex 2024-03-08T23:58:13.919 DEBUG:teuthology.orchestra.run.smithi102:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T23:58:13.937 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T23:58:13.937 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_1 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 662 Links: 1 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-08 23:57:57.490322533 +0000 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-08 23:52:12.975364633 +0000 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-08 23:52:12.975364633 +0000 2024-03-08T23:58:13.995 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-08 23:52:12.975364633 +0000 2024-03-08T23:58:13.996 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T23:58:14.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='client.24109 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi073=x;smithi102=y;smithi179=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: Saving service mgr spec with placement smithi073=x;smithi102=y;smithi179=z;count:3 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:13 smithi102 ceph-mon[36106]: Deploying daemon mgr.z on smithi179 2024-03-08T23:58:14.040 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-08T23:58:14.040 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-08T23:58:14.041 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000227004 s, 2.3 MB/s 2024-03-08T23:58:14.042 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T23:58:14.103 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_2 2024-03-08T23:58:14.162 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T23:58:14.162 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 682 Links: 1 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-08 23:57:57.491322514 +0000 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-08 23:52:13.309357792 +0000 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-08 23:52:13.309357792 +0000 2024-03-08T23:58:14.163 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-08 23:52:13.309357792 +0000 2024-03-08T23:58:14.163 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T23:58:14.233 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-08T23:58:14.233 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-08T23:58:14.233 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000249283 s, 2.1 MB/s 2024-03-08T23:58:14.234 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T23:58:14.292 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_3 2024-03-08T23:58:14.348 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-08 23:57:57.491322514 +0000 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-08 23:52:13.642350971 +0000 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-08 23:52:13.642350971 +0000 2024-03-08T23:58:14.349 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-08 23:52:13.642350971 +0000 2024-03-08T23:58:14.349 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T23:58:14.415 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-08T23:58:14.415 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-08T23:58:14.415 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000199335 s, 2.6 MB/s 2024-03-08T23:58:14.416 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T23:58:14.472 DEBUG:teuthology.orchestra.run.smithi102:> stat /dev/vg_nvme/lv_4 2024-03-08T23:58:14.527 INFO:teuthology.orchestra.run.smithi102.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Access: 2024-03-08 23:57:57.491322514 +0000 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Modify: 2024-03-08 23:52:13.980344048 +0000 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout:Change: 2024-03-08 23:52:13.980344048 +0000 2024-03-08T23:58:14.528 INFO:teuthology.orchestra.run.smithi102.stdout: Birth: 2024-03-08 23:52:13.980344048 +0000 2024-03-08T23:58:14.528 DEBUG:teuthology.orchestra.run.smithi102:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T23:58:14.592 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:14 smithi102 ceph-mon[36106]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records in 2024-03-08T23:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:1+0 records out 2024-03-08T23:58:14.594 INFO:teuthology.orchestra.run.smithi102.stderr:512 bytes copied, 0.000215692 s, 2.4 MB/s 2024-03-08T23:58:14.595 DEBUG:teuthology.orchestra.run.smithi102:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T23:58:14.651 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2024-03-08T23:58:14.651 DEBUG:teuthology.orchestra.run.smithi179:> dd if=/scratch_devs of=/dev/stdout 2024-03-08T23:58:14.667 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-08T23:58:14.668 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_1 2024-03-08T23:58:14.720 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:14 smithi179 ceph-mon[36637]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 664 Links: 1 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-08 23:58:14.590566239 +0000 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-08 23:52:16.065547055 +0000 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-08 23:52:16.065547055 +0000 2024-03-08T23:58:14.725 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-08 23:52:16.065547055 +0000 2024-03-08T23:58:14.725 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-08T23:58:14.795 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-08T23:58:14.795 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-08T23:58:14.795 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000252179 s, 2.0 MB/s 2024-03-08T23:58:14.796 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-08T23:58:14.853 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_2 2024-03-08T23:58:14.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:14 smithi073 ceph-mon[32309]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 683 Links: 1 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-08 23:58:14.590566239 +0000 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-08 23:52:16.410539587 +0000 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-08 23:52:16.410539587 +0000 2024-03-08T23:58:14.908 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-08 23:52:16.410539587 +0000 2024-03-08T23:58:14.909 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-08T23:58:14.972 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:14 smithi179 podman[38311]: 2024-03-08 23:58:14.898940535 +0000 UTC m=+0.127692564 container create b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-08T23:58:14.974 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-08T23:58:14.975 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-08T23:58:14.975 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000203302 s, 2.5 MB/s 2024-03-08T23:58:14.976 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-08T23:58:15.032 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_3 2024-03-08T23:58:15.086 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-08T23:58:15.086 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 701 Links: 1 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-08 23:58:14.591566219 +0000 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-08 23:52:16.737532508 +0000 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-08 23:52:16.737532508 +0000 2024-03-08T23:58:15.087 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-08 23:52:16.737532508 +0000 2024-03-08T23:58:15.087 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-08T23:58:15.153 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-08T23:58:15.153 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-08T23:58:15.153 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000200364 s, 2.6 MB/s 2024-03-08T23:58:15.155 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-08T23:58:15.212 DEBUG:teuthology.orchestra.run.smithi179:> stat /dev/vg_nvme/lv_4 2024-03-08T23:58:15.223 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 podman[38311]: 2024-03-08 23:58:15.126694244 +0000 UTC m=+0.355446301 container init b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-08T23:58:15.223 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 podman[38311]: 2024-03-08 23:58:15.129780771 +0000 UTC m=+0.358532815 container start b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-08T23:58:15.223 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 ceph-mgr[38444]: -- 172.21.15.179:0/2917191743 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d7dbc2a5a0 con 0x55d7dbbfe800 2024-03-08T23:58:15.224 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 bash[38311]: b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 2024-03-08T23:58:15.238 INFO:teuthology.orchestra.run.smithi179.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-08T23:58:15.238 INFO:teuthology.orchestra.run.smithi179.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Device: 5h/5d Inode: 721 Links: 1 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Context: system_u:object_r:device_t:s0 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Access: 2024-03-08 23:58:14.591566219 +0000 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Modify: 2024-03-08 23:52:17.058525559 +0000 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout:Change: 2024-03-08 23:52:17.058525559 +0000 2024-03-08T23:58:15.239 INFO:teuthology.orchestra.run.smithi179.stdout: Birth: 2024-03-08 23:52:17.058525559 +0000 2024-03-08T23:58:15.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-08T23:58:15.306 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records in 2024-03-08T23:58:15.307 INFO:teuthology.orchestra.run.smithi179.stderr:1+0 records out 2024-03-08T23:58:15.307 INFO:teuthology.orchestra.run.smithi179.stderr:512 bytes copied, 0.000216233 s, 2.4 MB/s 2024-03-08T23:58:15.308 DEBUG:teuthology.orchestra.run.smithi179:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-08T23:58:15.363 INFO:tasks.cephadm:Deploying osd.0 on smithi073 with /dev/vg_nvme/lv_4... 2024-03-08T23:58:15.364 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T23:58:15.582 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 systemd[1]: Started Ceph mgr.z for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:58:15.601 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:58:16.082 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:15.693+0000 7fce60d9c200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:58:16.082 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:15.785+0000 7fce60d9c200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:16.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: Deploying daemon mgr.y on smithi102 2024-03-08T23:58:16.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:16 smithi073 ceph-mon[32309]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: Deploying daemon mgr.y on smithi102 2024-03-08T23:58:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:16 smithi102 ceph-mon[36106]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:16.582 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:16.422+0000 7fce60d9c200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:58:16.582 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:16.513+0000 7fce60d9c200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:58:16.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: Deploying daemon mgr.y on smithi102 2024-03-08T23:58:16.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-mon[36637]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:17.082 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:16.655+0000 7fce60d9c200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:58:17.082 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:16.748+0000 7fce60d9c200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:58:17.082 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:16.830+0000 7fce60d9c200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:58:17.371 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:17.091+0000 7fce60d9c200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:58:17.372 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:17.200+0000 7fce60d9c200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:58:17.372 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:17.283+0000 7fce60d9c200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:58:17.372 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:17.376+0000 7fce60d9c200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:58:17.438 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 systemd[1]: Starting Ceph mgr.y for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116657 s, 899 MB/s 2024-03-08T23:58:17.517 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-08T23:58:17.731 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 podman[37510]: 2024-03-08 23:58:17.346016986 +0000 UTC m=+0.017794709 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-08T23:58:17.731 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 podman[37510]: 2024-03-08 23:58:17.447327531 +0000 UTC m=+0.119105260 container create 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-08T23:58:17.731 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 podman[37510]: 2024-03-08 23:58:17.736164141 +0000 UTC m=+0.407941859 container init 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-08T23:58:17.832 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:17.551+0000 7fce60d9c200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:58:18.009 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 podman[37510]: 2024-03-08 23:58:17.738697253 +0000 UTC m=+0.410474962 container start 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-08T23:58:18.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 ceph-mgr[37529]: -- 172.21.15.102:0/137264278 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561d4babe5a0 con 0x561d4ba93000 2024-03-08T23:58:18.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 bash[37510]: 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 2024-03-08T23:58:18.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:17 smithi102 systemd[1]: Started Ceph mgr.y for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-08T23:58:18.235 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_4 2024-03-08T23:58:18.466 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:58:18.760 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:18.284+0000 7fdc1286d200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-08T23:58:18.760 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:18.381+0000 7fdc1286d200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-08T23:58:18.832 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:18.539+0000 7fce60d9c200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:58:18.832 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:18.683+0000 7fce60d9c200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:58:19.249 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:19.031+0000 7fce60d9c200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:58:19.249 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:19.126+0000 7fce60d9c200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:19.250 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:19.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:19.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:19.261 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.056+0000 7fdc1286d200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-08T23:58:19.261 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.155+0000 7fdc1286d200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-08T23:58:19.512 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:19.321+0000 7fce60d9c200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:58:19.512 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:19.517+0000 7fce60d9c200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:58:19.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.295+0000 7fdc1286d200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-08T23:58:19.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.388+0000 7fdc1286d200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-08T23:58:19.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.479+0000 7fdc1286d200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-08T23:58:19.738 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.742+0000 7fdc1286d200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-08T23:58:20.009 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.848+0000 7fdc1286d200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-08T23:58:20.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:19.930+0000 7fdc1286d200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-08T23:58:20.058 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:19.801+0000 7fce60d9c200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:58:20.058 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:20.062+0000 7fce60d9c200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 -- 172.21.15.73:0/1244935902 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 msgr2=0x7f68581033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 --2- 172.21.15.73:0/1244935902 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f68581033f0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f6848009990 tx=0x7f6848009ca0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 -- 172.21.15.73:0/1244935902 shutdown_connections 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 --2- 172.21.15.73:0/1244935902 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858103930 0x7f685810a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 --2- 172.21.15.73:0/1244935902 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858101e10 0x7f6858102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 --2- 172.21.15.73:0/1244935902 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f68581033f0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 -- 172.21.15.73:0/1244935902 >> 172.21.15.73:0/1244935902 conn(0x7f68580fd390 msgr2=0x7f68580ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 -- 172.21.15.73:0/1244935902 shutdown_connections 2024-03-08T23:58:20.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.105+0000 7f685f39a700 1 -- 172.21.15.73:0/1244935902 wait complete. 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 Processor -- start 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 -- start start 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858101e10 0x7f6858197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858103930 0x7f685819f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685810d9e0 con 0x7f6858102f70 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685810d860 con 0x7f6858101e10 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685f39a700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f685810db60 con 0x7f6858103930 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42252/0 (socket says 172.21.15.73:42252) 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 -- 172.21.15.73:0/1213001212 learned_addr learned my addr 172.21.15.73:0/1213001212 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685d136700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858101e10 0x7f6858197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:20.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685d937700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858103930 0x7f685819f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 -- 172.21.15.73:0/1213001212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858103930 msgr2=0x7f685819f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858103930 0x7f685819f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 -- 172.21.15.73:0/1213001212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858101e10 msgr2=0x7f6858197fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858101e10 0x7f6858197fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.106+0000 7f685c935700 1 -- 172.21.15.73:0/1213001212 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685819fc20 con 0x7f6858102f70 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f685c935700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68480047e0 tx=0x7f6848009580 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6848004830 con 0x7f6858102f70 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6848003c00 con 0x7f6858102f70 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f685819feb0 con 0x7f6858102f70 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68480053a0 con 0x7f6858102f70 2024-03-08T23:58:20.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.107+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68581a0250 con 0x7f6858102f70 2024-03-08T23:58:20.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 13) v1 ==== 46162+0+0 (secure 0 0 0) 0x7f6848029060 con 0x7f6858102f70 2024-03-08T23:58:20.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f684e7fc700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 0x7f684403b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:20.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f684804dd60 con 0x7f6858102f70 2024-03-08T23:58:20.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f685d136700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 0x7f684403b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:20.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f685802d830 con 0x7f6858102f70 2024-03-08T23:58:20.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.108+0000 7f685d136700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 0x7f684403b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6850009550 tx=0x7f68500091f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:20.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.115+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6848013070 con 0x7f6858102f70 2024-03-08T23:58:20.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:20.334+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f6858062220 con 0x7f6844039200 2024-03-08T23:58:20.332 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:20.294+0000 7fce60d9c200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:58:20.505 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:20.021+0000 7fdc1286d200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-08T23:58:20.505 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:20.192+0000 7fdc1286d200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T23:58:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: Reconfiguring daemon mgr.x on smithi073 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:20.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: Reconfiguring daemon mgr.x on smithi073 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:20.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.891 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: Reconfiguring mgr.x (unknown last config time)... 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: Reconfiguring daemon mgr.x on smithi073 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:20.892 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:21.332 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:21.085+0000 7fce60d9c200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:58:21.509 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:21.144+0000 7fdc1286d200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-08T23:58:21.510 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:21.271+0000 7fdc1286d200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-08T23:58:21.832 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:21.655+0000 7fce60d9c200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:58:21.832 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:21.752+0000 7fce60d9c200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:21.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.867 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:21.598+0000 7fdc1286d200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:21.684+0000 7fdc1286d200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:21.871+0000 7fdc1286d200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:21.868 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.888 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='client.14217 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:21.889 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:22.259 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:22.048+0000 7fdc1286d200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-08T23:58:22.332 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 08 23:58:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-08T23:58:21.964+0000 7fce60d9c200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:58:22.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:22.547+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 14) v1 ==== 91698+0+0 (secure 0 0 0) 0x7f6848004070 con 0x7f6858102f70 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:22.322+0000 7fdc1286d200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: Standby manager daemon z started 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:58:22.570 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: Standby manager daemon z started 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:58:22.625 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:22 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: Standby manager daemon z started 2024-03-08T23:58:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-08T23:58:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-08T23:58:22.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:22 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/1304359246' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:23.009 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:22.574+0000 7fdc1286d200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-08T23:58:23.010 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:22.802+0000 7fdc1286d200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-08T23:58:23.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-08T23:58:23.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:58:23.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]: dispatch 2024-03-08T23:58:23.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]': finished 2024-03-08T23:58:23.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T23:58:23.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:23 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:23.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-08T23:58:23.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:58:23.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]: dispatch 2024-03-08T23:58:23.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]': finished 2024-03-08T23:58:23.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T23:58:23.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:23 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:24.009 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:23.580+0000 7fdc1286d200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: mgrmap e14: x(active, since 78s), standbys: z 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]: dispatch 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3929208542' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "23a8d6f6-f2b8-449c-8948-2a7287b837a9"}]': finished 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: osdmap e5: 1 total, 0 up, 1 in 2024-03-08T23:58:24.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:23 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:24.433 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:24.135+0000 7fdc1286d200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-08T23:58:24.433 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:24.230+0000 7fdc1286d200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-08T23:58:24.434 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-08T23:58:24.437+0000 7fdc1286d200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-08T23:58:24.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:24.555+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f684800f460 con 0x7f6858102f70 2024-03-08T23:58:24.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2668155445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: Standby manager daemon y started 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:58:24.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:24 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:24.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2668155445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:24.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:24.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: Standby manager daemon y started 2024-03-08T23:58:24.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:58:24.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:24.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:58:24.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:24 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2668155445' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: Standby manager daemon y started 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-08T23:58:24.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-08T23:58:24.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:24 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3157717249' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-08T23:58:25.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:25 smithi102 ceph-mon[36106]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-08T23:58:25.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:25 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:58:25.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:25 smithi179 ceph-mon[36637]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-08T23:58:25.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:25 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:58:25.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:25 smithi073 ceph-mon[32309]: mgrmap e15: x(active, since 80s), standbys: z, y 2024-03-08T23:58:25.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:25 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-08T23:58:26.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:26 smithi179 ceph-mon[36637]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:26.834 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:26 smithi073 ceph-mon[32309]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:27.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:26 smithi102 ceph-mon[36106]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:29.118 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:28 smithi073 ceph-mon[32309]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:29.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:28 smithi102 ceph-mon[36106]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:28 smithi179 ceph-mon[36637]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:31.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:30 smithi073 ceph-mon[32309]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:31.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T23:58:31.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:31.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:30 smithi102 ceph-mon[36106]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:31.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T23:58:31.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:30 smithi179 ceph-mon[36637]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-08T23:58:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:32.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:31 smithi073 ceph-mon[32309]: Deploying daemon osd.0 on smithi073 2024-03-08T23:58:32.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:31 smithi102 ceph-mon[36106]: Deploying daemon osd.0 on smithi073 2024-03-08T23:58:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:31 smithi179 ceph-mon[36637]: Deploying daemon osd.0 on smithi073 2024-03-08T23:58:33.096 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:32 smithi073 ceph-mon[32309]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:32 smithi102 ceph-mon[36106]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:32 smithi179 ceph-mon[36637]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:35.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:34 smithi073 ceph-mon[32309]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:35.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:34 smithi102 ceph-mon[36106]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:35.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:34 smithi179 ceph-mon[36637]: pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:37.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:36 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:36 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:36 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:37.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:36 smithi102 ceph-mon[36106]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:37.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:36 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:36 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:36 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:37.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:36 smithi179 ceph-mon[36637]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:37.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:37.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:37.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:36 smithi073 ceph-mon[32309]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:39.050 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:39.050 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.051 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.051 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:39.051 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:39.051 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:39.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:39.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:39.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.351+0000 7f684e7fc700 1 -- 172.21.15.73:0/1213001212 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6858062220 con 0x7f6844039200 2024-03-08T23:58:39.347 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 0 on host 'smithi073' 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 msgr2=0x7f684403b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 0x7f684403b6c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f6850009550 tx=0x7f68500091f0 comp rx=0 tx=0).stop 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 msgr2=0x7f6858198500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f68480047e0 tx=0x7f6848009580 comp rx=0 tx=0).stop 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 shutdown_connections 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6844039200 0x7f684403b6c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:39.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6858103930 0x7f685819f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:39.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6858101e10 0x7f6858197fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:39.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.353+0000 7f685f39a700 1 --2- 172.21.15.73:0/1213001212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6858102f70 0x7f6858198500 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:39.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.354+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 >> 172.21.15.73:0/1213001212 conn(0x7f68580fd390 msgr2=0x7f68581062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:39.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.354+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 shutdown_connections 2024-03-08T23:58:39.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:58:39.354+0000 7f685f39a700 1 -- 172.21.15.73:0/1213001212 wait complete. 2024-03-08T23:58:40.012 DEBUG:teuthology.orchestra.run.smithi073:osd.0> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.0.service 2024-03-08T23:58:40.015 INFO:tasks.cephadm:Deploying osd.1 on smithi102 with /dev/vg_nvme/lv_4... 2024-03-08T23:58:40.015 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T23:58:40.267 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:40 smithi073 ceph-mon[32309]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.667 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:40 smithi102 ceph-mon[36106]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:40.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:40.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:58:40.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-08T23:58:40.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:40 smithi179 ceph-mon[36637]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:41.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:41 smithi102 ceph-mon[36106]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T23:58:41.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:41 smithi102 ceph-mon[36106]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T23:58:41.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:41 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:41.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:41 smithi102 ceph-mon[36106]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:41.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:41 smithi102 ceph-mon[36106]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:41.547 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-mon[32309]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T23:58:41.547 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-mon[32309]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T23:58:41.547 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:41.547 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-mon[32309]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:41.547 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-mon[32309]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:41.547 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0[46677]: 2024-03-08T23:58:41.361+0000 7f96f0a25700 -1 osd.0 0 waiting for initial osdmap 2024-03-08T23:58:41.547 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:58:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0[46677]: 2024-03-08T23:58:41.364+0000 7f96ec818700 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:58:41.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:41 smithi179 ceph-mon[36637]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-08T23:58:41.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:41 smithi179 ceph-mon[36637]: osdmap e6: 1 total, 0 up, 1 in 2024-03-08T23:58:41.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:41 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:41.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:41 smithi179 ceph-mon[36637]: from='osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:41.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:41 smithi179 ceph-mon[36637]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:58:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T23:58:42.110 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T23:58:42.111 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-08T23:58:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-08T23:58:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0128568 s, 816 MB/s 2024-03-08T23:58:42.111 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-08T23:58:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:42 smithi073 ceph-mon[32309]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:58:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:42 smithi073 ceph-mon[32309]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T23:58:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:42 smithi073 ceph-mon[32309]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:42.735 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:42 smithi102 ceph-mon[36106]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:58:42.735 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:42 smithi102 ceph-mon[36106]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T23:58:42.735 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.735 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.735 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:42 smithi102 ceph-mon[36106]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:42.763 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_4 2024-03-08T23:58:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:42 smithi179 ceph-mon[36637]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:58:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:42 smithi179 ceph-mon[36637]: osdmap e7: 1 total, 0 up, 1 in 2024-03-08T23:58:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:42 smithi179 ceph-mon[36637]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-08T23:58:43.001 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:58:43.376 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] boot 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:43.377 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-08T23:58:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-08T23:58:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] boot 2024-03-08T23:58:43.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:43.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.734 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-08T23:58:43.734 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: osd.0 [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] boot 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: osdmap e8: 1 total, 1 up, 1 in 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:58:43.735 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:58:44.478 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:44 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi073 2024-03-08T23:58:44.478 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:44 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 17088M 2024-03-08T23:58:44.478 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:44 smithi102 ceph-mon[36106]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T23:58:44.478 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:44 smithi102 ceph-mon[36106]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 -- 172.21.15.102:0/416969514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 msgr2=0x7f2ed8109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/416969514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed8109170 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f2ed000b0d0 tx=0x7f2ed000b3e0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 -- 172.21.15.102:0/416969514 shutdown_connections 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/416969514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8100d20 0x7f2ed8101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/416969514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed8109170 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/416969514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8101f20 0x7f2ed8102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 -- 172.21.15.102:0/416969514 >> 172.21.15.102:0/416969514 conn(0x7f2ed80fc2e0 msgr2=0x7f2ed80fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:58:44.504 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 -- 172.21.15.102:0/416969514 shutdown_connections 2024-03-08T23:58:44.505 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.508+0000 7f2edd0c3700 1 -- 172.21.15.102:0/416969514 wait complete. 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 Processor -- start 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 -- start start 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8100d20 0x7f2ed8198030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8101f20 0x7f2ed8198570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed810c960 con 0x7f2ed8101f20 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed810c7e0 con 0x7f2ed81028c0 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.509+0000 7f2edd0c3700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ed810cae0 con 0x7f2ed8100d20 2024-03-08T23:58:44.507 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:35844/0 (socket says 172.21.15.102:35844) 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 -- 172.21.15.102:0/940263872 learned_addr learned my addr 172.21.15.102:0/940263872 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed6d9d700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8100d20 0x7f2ed8198030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed659c700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8101f20 0x7f2ed8198570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 -- 172.21.15.102:0/940263872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8100d20 msgr2=0x7f2ed8198030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8100d20 0x7f2ed8198030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 -- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8101f20 msgr2=0x7f2ed8198570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8101f20 0x7f2ed8198570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 -- 172.21.15.102:0/940263872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ed819fc90 con 0x7f2ed81028c0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ed759e700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00062a0 tx=0x7f2ed0007d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed001f070 con 0x7f2ed81028c0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ed0004020 con 0x7f2ed81028c0 2024-03-08T23:58:44.508 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ed819ff20 con 0x7f2ed81028c0 2024-03-08T23:58:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ed0004e60 con 0x7f2ed81028c0 2024-03-08T23:58:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.510+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ed81a0450 con 0x7f2ed81028c0 2024-03-08T23:58:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.511+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ed8068ad0 con 0x7f2ed81028c0 2024-03-08T23:58:44.509 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.512+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f2ed00041d0 con 0x7f2ed81028c0 2024-03-08T23:58:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.513+0000 7f2ebffff700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 0x7f2ec00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:58:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.513+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f2ed000f020 con 0x7f2ed81028c0 2024-03-08T23:58:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.513+0000 7f2ed6d9d700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 0x7f2ec00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:58:44.510 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.514+0000 7f2ed6d9d700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 0x7f2ec00a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8101d80 tx=0x7f2ecc00a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:58:44.513 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.518+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2ed8068ad0 con 0x7f2ed81028c0 2024-03-08T23:58:44.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:44 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi073 2024-03-08T23:58:44.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:44 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 17088M 2024-03-08T23:58:44.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:44 smithi073 ceph-mon[32309]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T23:58:44.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:44 smithi073 ceph-mon[32309]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:44.717 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:58:44.721+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2ed806ac30 con 0x7f2ec00a2800 2024-03-08T23:58:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:44 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi073 2024-03-08T23:58:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:44 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 17088M 2024-03-08T23:58:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:44 smithi179 ceph-mon[36637]: osdmap e9: 1 total, 1 up, 1 in 2024-03-08T23:58:44.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:44 smithi179 ceph-mon[36637]: pgmap v49: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:45.589 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:45 smithi102 ceph-mon[36106]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:45.589 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:45.589 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:45.589 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:45.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:45 smithi073 ceph-mon[32309]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:45.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:45.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:45.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:45 smithi179 ceph-mon[36637]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:58:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:58:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:58:45.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:46.636 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:46 smithi102 ceph-mon[36106]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:46.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:46 smithi073 ceph-mon[32309]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:46.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:46 smithi179 ceph-mon[36637]: pgmap v50: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:47.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:47 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/534879488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:47 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:47 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]': finished 2024-03-08T23:58:47.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:47 smithi073 ceph-mon[32309]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T23:58:47.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:47 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:58:47.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:47 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/534879488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:47 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:47 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]': finished 2024-03-08T23:58:47.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:47 smithi102 ceph-mon[36106]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T23:58:47.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:47 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:58:47.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:47 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/534879488' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:47 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]: dispatch 2024-03-08T23:58:47.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:47 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c4e798b8-2a59-4ab9-8e1b-687cb0a3099d"}]': finished 2024-03-08T23:58:47.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:47 smithi179 ceph-mon[36637]: osdmap e10: 2 total, 1 up, 2 in 2024-03-08T23:58:47.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:47 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:58:48.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:48 smithi073 ceph-mon[32309]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:48.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:48 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/2189515965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:48.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:48 smithi102 ceph-mon[36106]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:48.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:48 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/2189515965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:48.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:48 smithi179 ceph-mon[36637]: pgmap v52: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:48.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:48 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/2189515965' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:58:51.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:50 smithi073 ceph-mon[32309]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:51.207 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:50 smithi102 ceph-mon[36106]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:50 smithi179 ceph-mon[36637]: pgmap v53: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:53.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:52 smithi073 ceph-mon[32309]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:53.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:52 smithi102 ceph-mon[36106]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:53.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:52 smithi179 ceph-mon[36637]: pgmap v54: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:55.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:54 smithi102 ceph-mon[36106]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:55.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T23:58:55.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:54 smithi179 ceph-mon[36637]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T23:58:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:54 smithi073 ceph-mon[32309]: pgmap v55: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-08T23:58:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:58:56.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:55 smithi102 ceph-mon[36106]: Deploying daemon osd.1 on smithi102 2024-03-08T23:58:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:55 smithi179 ceph-mon[36637]: Deploying daemon osd.1 on smithi102 2024-03-08T23:58:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:55 smithi073 ceph-mon[32309]: Deploying daemon osd.1 on smithi102 2024-03-08T23:58:56.985 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:56 smithi102 ceph-mon[36106]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:57.303 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:56 smithi073 ceph-mon[32309]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:57.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:56 smithi179 ceph-mon[36637]: pgmap v56: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:58.993 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:58:58 smithi102 ceph-mon[36106]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:58:58 smithi179 ceph-mon[36637]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:58:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:58:58 smithi073 ceph-mon[32309]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:00 smithi179 ceph-mon[36637]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:00 smithi073 ceph-mon[32309]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:00.904 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.904 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:00.904 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:00.905 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:00 smithi102 ceph-mon[36106]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:03.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:03.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:03.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:02 smithi179 ceph-mon[36637]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.096 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:02 smithi102 ceph-mon[36106]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:03.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:02 smithi073 ceph-mon[32309]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:03.179 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.183+0000 7f2ebffff700 1 -- 172.21.15.102:0/940263872 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2ed806ac30 con 0x7f2ec00a2800 2024-03-08T23:59:03.179 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 1 on host 'smithi102' 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 msgr2=0x7f2ec00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 0x7f2ec00a4cc0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f2ed8101d80 tx=0x7f2ecc00a380 comp rx=0 tx=0).stop 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 msgr2=0x7f2ed819f5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2ed00062a0 tx=0x7f2ed0007d90 comp rx=0 tx=0).stop 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 shutdown_connections 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2ec00a2800 0x7f2ec00a4cc0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:03.181 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ed81028c0 0x7f2ed819f5f0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:03.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ed8100d20 0x7f2ed8198030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:03.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 --2- 172.21.15.102:0/940263872 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ed8101f20 0x7f2ed8198570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:03.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 >> 172.21.15.102:0/940263872 conn(0x7f2ed80fc2e0 msgr2=0x7f2ed806a520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:03.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.185+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 shutdown_connections 2024-03-08T23:59:03.182 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:03.186+0000 7f2edd0c3700 1 -- 172.21.15.102:0/940263872 wait complete. 2024-03-08T23:59:03.774 DEBUG:teuthology.orchestra.run.smithi102:osd.1> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.1.service 2024-03-08T23:59:03.776 INFO:tasks.cephadm:Deploying osd.2 on smithi179 with /dev/vg_nvme/lv_4... 2024-03-08T23:59:03.776 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-08T23:59:04.022 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:04 smithi073 ceph-mon[32309]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:04.441 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.442 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.442 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:04.442 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.442 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.442 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:04 smithi179 ceph-mon[36637]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-08T23:59:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:04 smithi102 ceph-mon[36106]: pgmap v60: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:05.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:05 smithi179 ceph-mon[36637]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T23:59:05.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:05 smithi179 ceph-mon[36637]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:05 smithi179 ceph-mon[36637]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T23:59:05.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:05.330 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:05 smithi179 ceph-mon[36637]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:05 smithi073 ceph-mon[32309]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T23:59:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:05 smithi073 ceph-mon[32309]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:05 smithi073 ceph-mon[32309]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T23:59:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:05 smithi073 ceph-mon[32309]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.426 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-mon[36106]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-08T23:59:05.426 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-mon[36106]: from='osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.426 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-mon[36106]: osdmap e11: 2 total, 1 up, 2 in 2024-03-08T23:59:05.426 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:05.426 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-mon[36106]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-08T23:59:05.857 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-08T23:59:05.880 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-08T23:59:05.880 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-08T23:59:05.881 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-08T23:59:05.881 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0111369 s, 942 MB/s 2024-03-08T23:59:05.881 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-08T23:59:06.119 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1[40730]: 2024-03-08T23:59:05.873+0000 7ff01efc1700 -1 osd.1 0 waiting for initial osdmap 2024-03-08T23:59:06.119 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1[40730]: 2024-03-08T23:59:05.878+0000 7ff016dac700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: from='osd.1 ' entity='osd.1' 2024-03-08T23:59:06.409 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: from='osd.1 ' entity='osd.1' 2024-03-08T23:59:06.421 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.435 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_4 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: osdmap e12: 2 total, 1 up, 2 in 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: from='osd.1 ' entity='osd.1' 2024-03-08T23:59:06.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:06.677 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi102 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 17080M 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] boot 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T23:59:07.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi102 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 17080M 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] boot 2024-03-08T23:59:07.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T23:59:07.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi102 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 17080M 2024-03-08T23:59:07.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: osd.1 [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] boot 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: osdmap e13: 2 total, 2 up, 2 in 2024-03-08T23:59:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-08T23:59:08.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 -- 172.21.15.179:0/3955690494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684103b30 msgr2=0x7f868410a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:08.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 --2- 172.21.15.179:0/3955690494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684103b30 0x7f868410a3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f867c00b0a0 tx=0x7f867c00b3b0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 -- 172.21.15.179:0/3955690494 shutdown_connections 2024-03-08T23:59:08.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 --2- 172.21.15.179:0/3955690494 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684100110 0x7f8684100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 --2- 172.21.15.179:0/3955690494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684103b30 0x7f868410a3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 --2- 172.21.15.179:0/3955690494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103190 0x7f86841035f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 -- 172.21.15.179:0/3955690494 >> 172.21.15.179:0/3955690494 conn(0x7f86840fba30 msgr2=0x7f86840fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 -- 172.21.15.179:0/3955690494 shutdown_connections 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.127+0000 7f8689fe6700 1 -- 172.21.15.179:0/3955690494 wait complete. 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 Processor -- start 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 -- start start 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 0x7f8684101140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103b30 0x7f8684101680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:08.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868410dac0 con 0x7f8684103190 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868410d940 con 0x7f8684103b30 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f8689fe6700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f868410dc40 con 0x7f8684100110 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f86837fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f86837fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:40270/0 (socket says 172.21.15.179:40270) 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.128+0000 7f86837fe700 1 -- 172.21.15.179:0/1190673140 learned_addr learned my addr 172.21.15.179:0/1190673140 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8683fff700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103b30 0x7f8684101680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8682ffd700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 0x7f8684101140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 -- 172.21.15.179:0/1190673140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103b30 msgr2=0x7f8684101680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103b30 0x7f8684101680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 -- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 msgr2=0x7f8684101140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 0x7f8684101140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:08.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 -- 172.21.15.179:0/1190673140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8684101f30 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8682ffd700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 0x7f8684101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f86837fe700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f867400c970 tx=0x7f867400cd30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674004220 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8674007d80 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86841a21e0 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8674003e50 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.129+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86841a27a0 con 0x7f8684100110 2024-03-08T23:59:08.128 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.130+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8684068aa0 con 0x7f8684100110 2024-03-08T23:59:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.131+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7f86740078a0 con 0x7f8684100110 2024-03-08T23:59:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.132+0000 7f8680ff9700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 0x7f866c0a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.132+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(13..13 src has 1..13) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f86740ccde0 con 0x7f8684100110 2024-03-08T23:59:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.132+0000 7f8682ffd700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 0x7f866c0a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:08.129 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.132+0000 7f8682ffd700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 0x7f866c0a4ee0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86780099d0 tx=0x7f8678008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:08.131 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.136+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8674014030 con 0x7f8684100110 2024-03-08T23:59:08.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:08 smithi179 ceph-mon[36637]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:08.338 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:08.343+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f8684062220 con 0x7f866c0a2a20 2024-03-08T23:59:08.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:08 smithi073 ceph-mon[32309]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:08.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:08 smithi102 ceph-mon[36106]: pgmap v65: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:09.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:09 smithi179 ceph-mon[36637]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T23:59:09.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:09 smithi179 ceph-mon[36637]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:09.221 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:09.222 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:09.222 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:09.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:09 smithi102 ceph-mon[36106]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T23:59:09.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:09 smithi102 ceph-mon[36106]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:09.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:09.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:09.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:09.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:09 smithi073 ceph-mon[32309]: osdmap e14: 2 total, 2 up, 2 in 2024-03-08T23:59:09.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:09 smithi073 ceph-mon[32309]: from='client.24157 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:09.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:09.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:09.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:10.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:10 smithi179 ceph-mon[36637]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:10.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:10 smithi102 ceph-mon[36106]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:10.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:10 smithi073 ceph-mon[32309]: pgmap v67: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:11 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/2350675410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:11 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:11 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]': finished 2024-03-08T23:59:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:11 smithi102 ceph-mon[36106]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T23:59:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:11.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:11 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/2350675410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:11 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:11 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]': finished 2024-03-08T23:59:11.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:11 smithi179 ceph-mon[36637]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T23:59:11.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:11.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:11 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/2350675410' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:11 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]: dispatch 2024-03-08T23:59:11.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:11 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "ba842f27-768e-4c79-827f-4e0ca1411d87"}]': finished 2024-03-08T23:59:11.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:11 smithi073 ceph-mon[32309]: osdmap e15: 3 total, 2 up, 3 in 2024-03-08T23:59:11.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:12 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/121848632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:12 smithi102 ceph-mon[36106]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:12.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:12 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/121848632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:12.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:12 smithi179 ceph-mon[36637]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:12.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:12 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/121848632' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:12.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:12 smithi073 ceph-mon[32309]: pgmap v69: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:15.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:14 smithi179 ceph-mon[36637]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:15.213 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:14 smithi102 ceph-mon[36106]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:15.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:14 smithi073 ceph-mon[32309]: pgmap v70: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:17.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:16 smithi179 ceph-mon[36637]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:16 smithi102 ceph-mon[36106]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:16 smithi073 ceph-mon[32309]: pgmap v71: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T23:59:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T23:59:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:18.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-08T23:59:18.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:18 smithi102 ceph-mon[36106]: Deploying daemon osd.2 on smithi179 2024-03-08T23:59:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:18 smithi102 ceph-mon[36106]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:19.293 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:18 smithi179 ceph-mon[36637]: Deploying daemon osd.2 on smithi179 2024-03-08T23:59:19.293 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:18 smithi179 ceph-mon[36637]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:19.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:18 smithi073 ceph-mon[32309]: Deploying daemon osd.2 on smithi179 2024-03-08T23:59:19.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:18 smithi073 ceph-mon[32309]: pgmap v72: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:21.170 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:20 smithi179 ceph-mon[36637]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:20 smithi102 ceph-mon[36106]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:21.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:20 smithi073 ceph-mon[32309]: pgmap v73: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:23.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:22 smithi179 ceph-mon[36637]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:23.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:23.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:22 smithi102 ceph-mon[36106]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:23.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:23.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:22 smithi073 ceph-mon[32309]: pgmap v74: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:23.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:23.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:24 smithi102 ceph-mon[36106]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:24 smithi179 ceph-mon[36637]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:25.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:24 smithi073 ceph-mon[32309]: pgmap v75: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:26.453 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 2 on host 'smithi179' 2024-03-08T23:59:26.453 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.458+0000 7f8680ff9700 1 -- 172.21.15.179:0/1190673140 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8684062220 con 0x7f866c0a2a20 2024-03-08T23:59:26.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.460+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 msgr2=0x7f866c0a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:26.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.460+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 0x7f866c0a4ee0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f86780099d0 tx=0x7f8678008040 comp rx=0 tx=0).stop 2024-03-08T23:59:26.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.460+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 msgr2=0x7f8684102af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:26.455 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.460+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f867400c970 tx=0x7f867400cd30 comp rx=0 tx=0).stop 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 shutdown_connections 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f866c0a2a20 0x7f866c0a4ee0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8684103b30 0x7f8684101680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8684100110 0x7f8684102af0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 --2- 172.21.15.179:0/1190673140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8684103190 0x7f8684101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 >> 172.21.15.179:0/1190673140 conn(0x7f86840fba30 msgr2=0x7f8684106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 shutdown_connections 2024-03-08T23:59:26.456 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-08T23:59:26.461+0000 7f8689fe6700 1 -- 172.21.15.179:0/1190673140 wait complete. 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.736 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:26.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: pgmap v76: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:26.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:27.046 DEBUG:teuthology.orchestra.run.smithi179:osd.2> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.2.service 2024-03-08T23:59:27.049 INFO:tasks.cephadm:Deploying osd.3 on smithi073 with /dev/vg_nvme/lv_3... 2024-03-08T23:59:27.049 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-08T23:59:27.296 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:59:27.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:27 smithi179 ceph-mon[36637]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:27.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:27 smithi179 ceph-mon[36637]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:27.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:27 smithi073 ceph-mon[32309]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:27.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:27 smithi073 ceph-mon[32309]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:27.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:27 smithi102 ceph-mon[36106]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:27.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:27 smithi102 ceph-mon[36106]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-08T23:59:28.696 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T23:59:28.696 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T23:59:28.696 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:28.696 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:28.696 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:28.697 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:28 smithi073 ceph-mon[32309]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:28.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T23:59:28.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T23:59:28.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:28.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:28.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:28.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-mon[36637]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:28.833 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2[41807]: 2024-03-08T23:59:28.532+0000 7f8da4e79700 -1 osd.2 0 waiting for initial osdmap 2024-03-08T23:59:28.833 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2[41807]: 2024-03-08T23:59:28.535+0000 7f8da0c6c700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:59:29.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-08T23:59:29.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: osdmap e16: 3 total, 2 up, 3 in 2024-03-08T23:59:29.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:29.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: from='osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:29.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-08T23:59:29.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:28 smithi102 ceph-mon[36106]: pgmap v78: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114083 s, 919 MB/s 2024-03-08T23:59:29.437 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-08T23:59:29.769 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:29 smithi073 ceph-mon[32309]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-08T23:59:29.769 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:29 smithi073 ceph-mon[32309]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T23:59:29.770 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:29 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:29.770 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:29 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:29.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:29 smithi179 ceph-mon[36637]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-08T23:59:29.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:29 smithi179 ceph-mon[36637]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T23:59:29.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:29 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:29.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:29 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:30.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:29 smithi102 ceph-mon[36106]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-08T23:59:30.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:29 smithi102 ceph-mon[36106]: osdmap e17: 3 total, 2 up, 3 in 2024-03-08T23:59:30.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:29 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:30.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:29 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:30.098 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_3 2024-03-08T23:59:30.340 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-08T23:59:30.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-08T23:59:30.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] boot 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi179 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:30.808 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-08T23:59:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] boot 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi179 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:30.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:31.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: osd.2 [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] boot 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: osdmap e18: 3 total, 3 up, 3 in 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: pgmap v81: 0 pgs: ; 0 B data, 57 MiB used, 179 GiB / 179 GiB avail 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi179 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:31.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:31.778 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:31 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 17080M 2024-03-08T23:59:31.778 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:31 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:31.778 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:31 smithi073 ceph-mon[32309]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T23:59:31.778 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:31 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:31.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:31 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 17080M 2024-03-08T23:59:31.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:31 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:31.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:31 smithi179 ceph-mon[36637]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T23:59:31.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:31 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:32.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:31 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 17080M 2024-03-08T23:59:32.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:31 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:32.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:31 smithi102 ceph-mon[36106]: osdmap e19: 3 total, 3 up, 3 in 2024-03-08T23:59:32.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:31 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-08T23:59:32.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- 172.21.15.73:0/910079822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 msgr2=0x7fb5ec100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:32.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/910079822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec100540 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8009990 tx=0x7fb5e8009ca0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- 172.21.15.73:0/910079822 shutdown_connections 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/910079822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec100540 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/910079822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec103b40 0x7fb5ec10a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/910079822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec1031a0 0x7fb5ec103600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- 172.21.15.73:0/910079822 >> 172.21.15.73:0/910079822 conn(0x7fb5ec0fba80 msgr2=0x7fb5ec0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- 172.21.15.73:0/910079822 shutdown_connections 2024-03-08T23:59:32.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- 172.21.15.73:0/910079822 wait complete. 2024-03-08T23:59:32.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 Processor -- start 2024-03-08T23:59:32.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.015+0000 7fb5f3efd700 1 -- start start 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec1031a0 0x7fb5ec198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec103b40 0x7fb5ec19f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec10da10 con 0x7fb5ec100120 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec10d890 con 0x7fb5ec1031a0 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5ec10db90 con 0x7fb5ec103b40 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:44104/0 (socket says 172.21.15.73:44104) 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 -- 172.21.15.73:0/698062301 learned_addr learned my addr 172.21.15.73:0/698062301 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1498700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec1031a0 0x7fb5ec198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f249a700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec103b40 0x7fb5ec19f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 -- 172.21.15.73:0/698062301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec103b40 msgr2=0x7fb5ec19f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:32.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec103b40 0x7fb5ec19f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 -- 172.21.15.73:0/698062301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec1031a0 msgr2=0x7fb5ec198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec1031a0 0x7fb5ec198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 -- 172.21.15.73:0/698062301 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5ec19fc60 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f1c99700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8000c00 tx=0x7fb5e80047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8003c70 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5e8004830 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5ec19fef0 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.016+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5e8005330 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.017+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5ec1a0290 con 0x7fb5ec100120 2024-03-08T23:59:32.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.017+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5ec02d830 con 0x7fb5ec100120 2024-03-08T23:59:32.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.018+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 15) v1 ==== 137234+0+0 (secure 0 0 0) 0x7fb5e80049e0 con 0x7fb5ec100120 2024-03-08T23:59:32.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.019+0000 7fb5e2ffd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 0x7fb5d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:32.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.019+0000 7fb5f1498700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 0x7fb5d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:32.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.019+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 3237+0+0 (secure 0 0 0) 0x7fb5e80ceae0 con 0x7fb5ec100120 2024-03-08T23:59:32.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.019+0000 7fb5f1498700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 0x7fb5d80a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc005950 tx=0x7fb5dc00b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:32.018 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.023+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb5e8013070 con 0x7fb5ec100120 2024-03-08T23:59:32.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:32.228+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5ec062220 con 0x7fb5d80a2800 2024-03-08T23:59:32.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:32 smithi073 sudo[51497]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:32.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:32 smithi073 sudo[51497]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:32.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:32 smithi073 sudo[51497]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:32.806 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:32 smithi073 sudo[51497]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:32.807 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:32.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:32.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T23:59:32.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:32.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:32.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:32.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:33.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-08T23:59:33.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: osdmap e20: 3 total, 3 up, 3 in 2024-03-08T23:59:33.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: pgmap v84: 1 pgs: 1 unknown; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:33.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:33.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:33.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:33.084 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:33 smithi073 sudo[51504]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:33.084 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:33 smithi073 sudo[51504]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:33.084 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:33 smithi073 sudo[51504]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:33.084 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 08 23:59:33 smithi073 sudo[51504]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:33.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:33.563+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb5e8003e10 con 0x7fb5ec100120 2024-03-08T23:59:33.571 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:33 smithi102 ceph-mon[36106]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:33.571 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:33 smithi102 ceph-mon[36106]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T23:59:33.572 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:33 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:33.572 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44721]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:33.572 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44721]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:33.572 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44721]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:33.572 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44721]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:33.806 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:33 smithi073 ceph-mon[32309]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:33.806 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:33 smithi073 ceph-mon[32309]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T23:59:33.806 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:33 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:33.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:33 smithi179 ceph-mon[36637]: from='client.14286 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:33.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:33 smithi179 ceph-mon[36637]: osdmap e21: 3 total, 3 up, 3 in 2024-03-08T23:59:33.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:33 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:34.009 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44724]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:34.009 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44724]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.009 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44724]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.009 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 08 23:59:33 smithi102 sudo[44724]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.086 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:33 smithi179 sudo[45437]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:34.086 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:33 smithi179 sudo[45437]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.087 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:33 smithi179 sudo[45437]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.087 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:33 smithi179 sudo[45437]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51551]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51551]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51551]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51551]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.582 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45440]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:34.582 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45440]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.582 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45440]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.582 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45440]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 ceph-mon[36637]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T23:59:34.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:34.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 ceph-mon[36637]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:34.896 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:34.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:34.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45443]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:34.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45443]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45443]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.897 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:34 smithi179 sudo[45443]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51573]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51573]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51573]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 sudo[51573]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 ceph-mon[32309]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 ceph-mon[32309]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:34.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:34 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:35.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:34 smithi102 ceph-mon[36106]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-08T23:59:35.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:35.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:34 smithi102 ceph-mon[36106]: pgmap v86: 1 pgs: 1 creating+activating; 0 B data, 84 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:35.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:35.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:34 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:35.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 sudo[45446]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:35.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 sudo[45446]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:35.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 sudo[45446]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:35.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 sudo[45446]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:35.627 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44727]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-08T23:59:35.627 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44727]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:35.627 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44727]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:35.627 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44727]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:35.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:35.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:35.904 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]': finished 2024-03-08T23:59:35.904 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T23:59:35.904 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:35.904 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:35 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44730]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44730]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44730]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 sudo[44730]: pam_unix(sudo:session): session closed for user root 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]: dispatch 2024-03-08T23:59:36.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]': finished 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:36.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:35 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/597455303' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b9390f11-f076-4591-b204-73187536054b"}]': finished 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: osdmap e22: 4 total, 3 up, 4 in 2024-03-08T23:59:36.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:36.084 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:35 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-08T23:59:36.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:36 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/90009352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:36.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:36 smithi073 ceph-mon[32309]: pgmap v88: 1 pgs: 1 creating+activating; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:37.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:36 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/90009352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:37.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:36 smithi102 ceph-mon[36106]: pgmap v88: 1 pgs: 1 creating+activating; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:37.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:36 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/90009352' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-08T23:59:37.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:36 smithi179 ceph-mon[36637]: pgmap v88: 1 pgs: 1 creating+activating; 0 B data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:39.116 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:38 smithi073 ceph-mon[32309]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:38 smithi102 ceph-mon[36106]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:38 smithi179 ceph-mon[36637]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:40 smithi102 ceph-mon[36106]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:41.308 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:40 smithi073 ceph-mon[32309]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:41.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:40 smithi179 ceph-mon[36637]: pgmap v90: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:43.240 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:42 smithi073 ceph-mon[32309]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:43.241 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T23:59:43.241 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:42 smithi102 ceph-mon[36106]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T23:59:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:42 smithi179 ceph-mon[36637]: pgmap v91: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-08T23:59:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:44.233 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:43 smithi179 ceph-mon[36637]: Deploying daemon osd.3 on smithi073 2024-03-08T23:59:44.243 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:43 smithi073 ceph-mon[32309]: Deploying daemon osd.3 on smithi073 2024-03-08T23:59:44.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:43 smithi102 ceph-mon[36106]: Deploying daemon osd.3 on smithi073 2024-03-08T23:59:45.243 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:44 smithi073 ceph-mon[32309]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:45.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:44 smithi102 ceph-mon[36106]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:45.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:44 smithi179 ceph-mon[36637]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:47.234 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:46 smithi073 ceph-mon[32309]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:46 smithi102 ceph-mon[36106]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:47.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:46 smithi179 ceph-mon[36637]: pgmap v93: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:49.208 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:48 smithi073 ceph-mon[32309]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:49.208 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.208 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.208 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:48 smithi102 ceph-mon[36106]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:48 smithi179 ceph-mon[36637]: pgmap v94: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:51.187 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:50 smithi073 ceph-mon[32309]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:50 smithi102 ceph-mon[36106]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:50 smithi179 ceph-mon[36637]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:51.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.883+0000 7fb5e2ffd700 1 -- 172.21.15.73:0/698062301 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fb5ec062220 con 0x7fb5d80a2800 2024-03-08T23:59:51.879 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 3 on host 'smithi073' 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 msgr2=0x7fb5d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 0x7fb5d80a4cc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fb5dc005950 tx=0x7fb5dc00b410 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 msgr2=0x7fb5ec198000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fb5e8000c00 tx=0x7fb5e80047b0 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 shutdown_connections 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5ec100120 0x7fb5ec198000 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5d80a2800 0x7fb5d80a4cc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5ec103b40 0x7fb5ec19f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 --2- 172.21.15.73:0/698062301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5ec1031a0 0x7fb5ec198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 >> 172.21.15.73:0/698062301 conn(0x7fb5ec0fba80 msgr2=0x7fb5ec106440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 shutdown_connections 2024-03-08T23:59:51.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-08T23:59:51.885+0000 7fb5f3efd700 1 -- 172.21.15.73:0/698062301 wait complete. 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.083 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:52.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-08T23:59:52.577 DEBUG:teuthology.orchestra.run.smithi073:osd.3> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.3.service 2024-03-08T23:59:52.579 INFO:tasks.cephadm:Deploying osd.4 on smithi102 with /dev/vg_nvme/lv_3... 2024-03-08T23:59:52.579 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-08T23:59:52.821 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:59:53.062 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:52 smithi073 ceph-mon[32309]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:53.063 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:52 smithi073 ceph-mon[32309]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T23:59:53.247 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:52 smithi102 ceph-mon[36106]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:53.248 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:52 smithi102 ceph-mon[36106]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T23:59:53.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:52 smithi179 ceph-mon[36637]: pgmap v96: 1 pgs: 1 active+clean; 577 KiB data, 86 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:53.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:52 smithi179 ceph-mon[36637]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-08T23:59:54.126 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-mon[32309]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T23:59:54.126 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-mon[32309]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T23:59:54.126 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-mon[32309]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:59:54.126 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:54.126 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-mon[32309]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:59:54.126 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-08T23:59:53.961+0000 7f0b5d8d8700 -1 osd.3 0 waiting for initial osdmap 2024-03-08T23:59:54.126 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 08 23:59:53 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-08T23:59:53.967+0000 7f0b596cb700 -1 osd.3 24 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-08T23:59:54.202 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:53 smithi102 ceph-mon[36106]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T23:59:54.202 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:53 smithi102 ceph-mon[36106]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T23:59:54.202 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:53 smithi102 ceph-mon[36106]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:59:54.202 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:53 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:54.202 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:53 smithi102 ceph-mon[36106]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:59:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:53 smithi179 ceph-mon[36637]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-08T23:59:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:53 smithi179 ceph-mon[36637]: osdmap e23: 4 total, 3 up, 4 in 2024-03-08T23:59:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:53 smithi179 ceph-mon[36637]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-08T23:59:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:53 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:53 smithi179 ceph-mon[36637]: from='osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.011219 s, 935 MB/s 2024-03-08T23:59:54.774 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] boot 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T23:59:55.142 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] boot 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T23:59:55.290 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 85 MiB used, 268 GiB / 268 GiB avail 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: osdmap e24: 4 total, 3 up, 4 in 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: osd.3 [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] boot 2024-03-08T23:59:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: osdmap e25: 4 total, 4 up, 4 in 2024-03-08T23:59:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-08T23:59:55.461 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_3 2024-03-08T23:59:55.699 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:56.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:56.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-08T23:59:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-08T23:59:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T23:59:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:56.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: osdmap e26: 4 total, 4 up, 4 in 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-08T23:59:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-08T23:59:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-08T23:59:57.094 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:56 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi073 2024-03-08T23:59:57.094 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:56 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 8544M 2024-03-08T23:59:57.094 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:56 smithi102 ceph-mon[36106]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:59:57.094 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:56 smithi102 ceph-mon[36106]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T23:59:57.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:56 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi073 2024-03-08T23:59:57.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:56 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 8544M 2024-03-08T23:59:57.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:56 smithi073 ceph-mon[32309]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:59:57.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:56 smithi073 ceph-mon[32309]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T23:59:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.336+0000 7fc410dae700 1 -- 172.21.15.102:0/2629145530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c100d40 msgr2=0x7fc40c101160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:57.333 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.336+0000 7fc410dae700 1 --2- 172.21.15.102:0/2629145530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c100d40 0x7fc40c101160 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fc4000099c0 tx=0x7fc400009cd0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 -- 172.21.15.102:0/2629145530 shutdown_connections 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 --2- 172.21.15.102:0/2629145530 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c1028e0 0x7fc40c109190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 --2- 172.21.15.102:0/2629145530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c100d40 0x7fc40c101160 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 --2- 172.21.15.102:0/2629145530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 0x7fc40c1023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 -- 172.21.15.102:0/2629145530 >> 172.21.15.102:0/2629145530 conn(0x7fc40c0fc300 msgr2=0x7fc40c0fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 -- 172.21.15.102:0/2629145530 shutdown_connections 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 -- 172.21.15.102:0/2629145530 wait complete. 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 Processor -- start 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 -- start start 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.337+0000 7fc410dae700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 0x7fc40c06b080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:57.334 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 0x7fc40c19f160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c10c920 con 0x7fc40c100d40 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c10c7a0 con 0x7fc40c1028e0 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc40c10caa0 con 0x7fc40c101f40 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:48332/0 (socket says 172.21.15.102:48332) 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 -- 172.21.15.102:0/2584703251 learned_addr learned my addr 172.21.15.102:0/2584703251 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40a59c700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 0x7fc40c06b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc409d9b700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 0x7fc40c19f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 -- 172.21.15.102:0/2584703251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 msgr2=0x7fc40c19f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 0x7fc40c19f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 -- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 msgr2=0x7fc40c06b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-08T23:59:57.335 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 0x7fc40c06b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 -- 172.21.15.102:0/2584703251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc40c1a1e90 con 0x7fc40c1028e0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40a59c700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 0x7fc40c06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc40ad9d700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc404007d70 tx=0x7fc40400a340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404016020 con 0x7fc40c1028e0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc40400ad40 con 0x7fc40c1028e0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc40c1a2180 con 0x7fc40c1028e0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc404011600 con 0x7fc40c1028e0 2024-03-08T23:59:57.336 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.338+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc40c1a2670 con 0x7fc40c1028e0 2024-03-08T23:59:57.337 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:56 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi073 2024-03-08T23:59:57.337 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:56 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 8544M 2024-03-08T23:59:57.337 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:56 smithi179 ceph-mon[36637]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-08T23:59:57.337 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:56 smithi179 ceph-mon[36637]: osdmap e27: 4 total, 4 up, 4 in 2024-03-08T23:59:57.337 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.339+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc40c068aa0 con 0x7fc40c1028e0 2024-03-08T23:59:57.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.340+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc404019020 con 0x7fc40c1028e0 2024-03-08T23:59:57.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.341+0000 7fc3fb7fe700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 0x7fc3f40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-08T23:59:57.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.341+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(27..27 src has 1..27) v4 ==== 3698+0+0 (secure 0 0 0) 0x7fc4040cdbf0 con 0x7fc40c1028e0 2024-03-08T23:59:57.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.341+0000 7fc40a59c700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 0x7fc3f40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-08T23:59:57.338 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.342+0000 7fc40a59c700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 0x7fc3f40a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc40000b580 tx=0x7fc400005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-08T23:59:57.339 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.345+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc404020020 con 0x7fc40c1028e0 2024-03-08T23:59:57.544 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-08T23:59:57.550+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fc40c062220 con 0x7fc3f40a2890 2024-03-08T23:59:58.236 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:57 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:58.236 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:57 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:58.236 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:57 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:58.236 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:57 smithi102 ceph-mon[36106]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:59:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:57 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:57 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:57 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:57 smithi179 ceph-mon[36637]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:59:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:57 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-08T23:59:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:57 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-08T23:59:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:57 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-08T23:59:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:57 smithi073 ceph-mon[32309]: osdmap e28: 4 total, 4 up, 4 in 2024-03-08T23:59:59.159 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:58 smithi102 ceph-mon[36106]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:59.159 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:58 smithi102 ceph-mon[36106]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-08T23:59:59.159 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 08 23:59:58 smithi102 ceph-mon[36106]: osdmap e29: 4 total, 4 up, 4 in 2024-03-08T23:59:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:58 smithi179 ceph-mon[36637]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:58 smithi179 ceph-mon[36637]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-08T23:59:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 08 23:59:58 smithi179 ceph-mon[36637]: osdmap e29: 4 total, 4 up, 4 in 2024-03-08T23:59:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:58 smithi073 ceph-mon[32309]: from='client.24215 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-08T23:59:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:58 smithi073 ceph-mon[32309]: pgmap v105: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 148 KiB/s, 0 objects/s recovering 2024-03-08T23:59:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 08 23:59:58 smithi073 ceph-mon[32309]: osdmap e29: 4 total, 4 up, 4 in 2024-03-09T00:00:01.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:01.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: overall HEALTH_OK 2024-03-09T00:00:01.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/2043003658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]': finished 2024-03-09T00:00:01.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: osdmap e30: 5 total, 4 up, 5 in 2024-03-09T00:00:01.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:01.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:00 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/1034478493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:01.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:01.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: overall HEALTH_OK 2024-03-09T00:00:01.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/2043003658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]': finished 2024-03-09T00:00:01.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: osdmap e30: 5 total, 4 up, 5 in 2024-03-09T00:00:01.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:01.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:00 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/1034478493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:01.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: pgmap v107: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:01.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: overall HEALTH_OK 2024-03-09T00:00:01.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/2043003658' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]: dispatch 2024-03-09T00:00:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "be6f4676-0213-47ac-806e-52fe6bddb9e2"}]': finished 2024-03-09T00:00:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: osdmap e30: 5 total, 4 up, 5 in 2024-03-09T00:00:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:00 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/1034478493' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:03.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:02 smithi102 ceph-mon[36106]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:03.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:02 smithi179 ceph-mon[36637]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:03.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:02 smithi073 ceph-mon[32309]: pgmap v109: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 113 MiB used, 357 GiB / 358 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:04.152 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:00:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:00:03.857+0000 7fb3c5462700 -1 calc_pg_upmaps abort due to max <= 0 2024-03-09T00:00:05.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:04 smithi102 ceph-mon[36106]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:04 smithi179 ceph-mon[36637]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:05.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:04 smithi073 ceph-mon[32309]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:07.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:06 smithi102 ceph-mon[36106]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:06 smithi179 ceph-mon[36637]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:06 smithi073 ceph-mon[32309]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:08.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-09T00:00:08.259 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:08.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-09T00:00:08.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:08.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-09T00:00:08.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:09.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:08 smithi102 ceph-mon[36106]: Deploying daemon osd.4 on smithi102 2024-03-09T00:00:09.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:08 smithi102 ceph-mon[36106]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:09.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:08 smithi179 ceph-mon[36637]: Deploying daemon osd.4 on smithi102 2024-03-09T00:00:09.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:08 smithi179 ceph-mon[36637]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:08 smithi073 ceph-mon[32309]: Deploying daemon osd.4 on smithi102 2024-03-09T00:00:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:08 smithi073 ceph-mon[32309]: pgmap v112: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:11.234 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:10 smithi102 ceph-mon[36106]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:10 smithi179 ceph-mon[36637]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:11.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:10 smithi073 ceph-mon[32309]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:13.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:12 smithi102 ceph-mon[36106]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:13.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:12 smithi179 ceph-mon[36637]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:13.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:12 smithi073 ceph-mon[32309]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:14.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:14.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:14 smithi102 ceph-mon[36106]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:14.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:14.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:14 smithi073 ceph-mon[32309]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:14 smithi179 ceph-mon[36637]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:16.909 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 4 on host 'smithi102' 2024-03-09T00:00:16.909 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.912+0000 7fc3fb7fe700 1 -- 172.21.15.102:0/2584703251 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fc40c062220 con 0x7fc3f40a2890 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 msgr2=0x7fc3f40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 0x7fc3f40a4d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc40000b580 tx=0x7fc400005fb0 comp rx=0 tx=0).stop 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 msgr2=0x7fc40c1a16b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fc404007d70 tx=0x7fc40400a340 comp rx=0 tx=0).stop 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 shutdown_connections 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc40c100d40 0x7fc40c06b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc3f40a2890 0x7fc3f40a4d50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:16.911 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc40c1028e0 0x7fc40c1a16b0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:16.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 --2- 172.21.15.102:0/2584703251 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc40c101f40 0x7fc40c19f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:16.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 >> 172.21.15.102:0/2584703251 conn(0x7fc40c0fc300 msgr2=0x7fc40c105240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:00:16.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 shutdown_connections 2024-03-09T00:00:16.912 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:00:16.915+0000 7fc410dae700 1 -- 172.21.15.102:0/2584703251 wait complete. 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:17.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:17 smithi102 ceph-mon[36106]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:17.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:17 smithi179 ceph-mon[36637]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:17 smithi073 ceph-mon[32309]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-09T00:00:17.649 DEBUG:teuthology.orchestra.run.smithi102:osd.4> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.4.service 2024-03-09T00:00:17.652 INFO:tasks.cephadm:Deploying osd.5 on smithi179 with /dev/vg_nvme/lv_3... 2024-03-09T00:00:17.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-09T00:00:17.923 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: osdmap e31: 5 total, 4 up, 5 in 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:19.038 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:18 smithi179 ceph-mon[36637]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:19.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-09T00:00:19.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: osdmap e31: 5 total, 4 up, 5 in 2024-03-09T00:00:19.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:19.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:18 smithi073 ceph-mon[32309]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.154 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: pgmap v117: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:19.154 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-09T00:00:19.154 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: from='osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.154 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: osdmap e31: 5 total, 4 up, 5 in 2024-03-09T00:00:19.155 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:19.155 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:18 smithi102 ceph-mon[36106]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:00:19.883 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4[47558]: 2024-03-09T00:00:19.609+0000 7f167c3a9700 -1 osd.4 0 waiting for initial osdmap 2024-03-09T00:00:19.883 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4[47558]: 2024-03-09T00:00:19.616+0000 7f1674194700 -1 osd.4 32 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.01114 s, 941 MB/s 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 2024-03-09T00:00:20.016 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-09T00:00:20.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:19 smithi073 ceph-mon[32309]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:00:20.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:19 smithi073 ceph-mon[32309]: osdmap e32: 5 total, 4 up, 5 in 2024-03-09T00:00:20.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:19 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:19 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:19 smithi073 ceph-mon[32309]: from='osd.4 ' entity='osd.4' 2024-03-09T00:00:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-mon[36106]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:00:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-mon[36106]: osdmap e32: 5 total, 4 up, 5 in 2024-03-09T00:00:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:19 smithi102 ceph-mon[36106]: from='osd.4 ' entity='osd.4' 2024-03-09T00:00:20.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:19 smithi179 ceph-mon[36637]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:00:20.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:19 smithi179 ceph-mon[36637]: osdmap e32: 5 total, 4 up, 5 in 2024-03-09T00:00:20.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:19 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:19 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:20.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:19 smithi179 ceph-mon[36637]: from='osd.4 ' entity='osd.4' 2024-03-09T00:00:20.650 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_3 2024-03-09T00:00:20.893 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:00:21.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] boot 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: osdmap e33: 5 total, 5 up, 5 in 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:21.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:21.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:00:21.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:00:21.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] boot 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: osdmap e33: 5 total, 5 up, 5 in 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:21.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:00:21.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:00:21.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 112 MiB used, 357 GiB / 358 GiB avail 2024-03-09T00:00:21.155 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: osd.4 [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] boot 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: osdmap e33: 5 total, 5 up, 5 in 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:21.156 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:22.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:21 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi102 2024-03-09T00:00:22.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:21 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 8540M 2024-03-09T00:00:22.190 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:21 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi102 2024-03-09T00:00:22.190 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:21 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 8540M 2024-03-09T00:00:22.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:21 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi102 2024-03-09T00:00:22.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:21 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 8540M 2024-03-09T00:00:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/141046916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74100d00 msgr2=0x7f8a74101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/141046916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74100d00 0x7f8a74101120 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f8a64009990 tx=0x7f8a64009ca0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/141046916 shutdown_connections 2024-03-09T00:00:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/141046916 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a741028a0 0x7f8a74109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/141046916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74100d00 0x7f8a74101120 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/141046916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a74101f00 0x7f8a74102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/141046916 >> 172.21.15.179:0/141046916 conn(0x7f8a740fc2e0 msgr2=0x7f8a740fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/141046916 shutdown_connections 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.408+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/141046916 wait complete. 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 Processor -- start 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 -- start start 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a74100d00 0x7f8a7406b0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a741028a0 0x7f8a7419f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7410c9c0 con 0x7f8a74100d00 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7410c840 con 0x7f8a741028a0 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7b8f3700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a7410cb40 con 0x7f8a74101f00 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:22.407 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:48892/0 (socket says 172.21.15.179:48892) 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 -- 172.21.15.179:0/1879400080 learned_addr learned my addr 172.21.15.179:0/1879400080 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a7968f700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a74100d00 0x7f8a7406b0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a79e90700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a741028a0 0x7f8a7419f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 -- 172.21.15.179:0/1879400080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a741028a0 msgr2=0x7f8a7419f520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a741028a0 0x7f8a7419f520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 -- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a74100d00 msgr2=0x7f8a7406b0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a74100d00 0x7f8a7406b0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.409+0000 7f8a78e8e700 1 -- 172.21.15.179:0/1879400080 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a7419fdf0 con 0x7f8a74101f00 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a78e8e700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b530 tx=0x7f8a7000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a70015020 con 0x7f8a74101f00 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8a70011d10 con 0x7f8a74101f00 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a741a00e0 con 0x7f8a74101f00 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8a7000f2e0 con 0x7f8a74101f00 2024-03-09T00:00:22.408 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.410+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a741a0650 con 0x7f8a74101f00 2024-03-09T00:00:22.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.411+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a74068aa0 con 0x7f8a74101f00 2024-03-09T00:00:22.409 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.412+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8a70018020 con 0x7f8a74101f00 2024-03-09T00:00:22.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.412+0000 7f8a6a7fc700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 0x7f8a600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:22.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.413+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f8a700cd120 con 0x7f8a74101f00 2024-03-09T00:00:22.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.413+0000 7f8a7968f700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 0x7f8a600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:22.410 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.413+0000 7f8a7968f700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 0x7f8a600a4d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a64003890 tx=0x7f8a64005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:00:22.413 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.417+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8a7001f020 con 0x7f8a74101f00 2024-03-09T00:00:22.615 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:22.620+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f8a74062220 con 0x7f8a600a2890 2024-03-09T00:00:23.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:22 smithi073 ceph-mon[32309]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:23.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:22 smithi073 ceph-mon[32309]: osdmap e34: 5 total, 5 up, 5 in 2024-03-09T00:00:23.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:23.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:23.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:23.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:22 smithi179 ceph-mon[36637]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:23.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:22 smithi179 ceph-mon[36637]: osdmap e34: 5 total, 5 up, 5 in 2024-03-09T00:00:23.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:23.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:23.219 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:23.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:22 smithi102 ceph-mon[36106]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:23.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:22 smithi102 ceph-mon[36106]: osdmap e34: 5 total, 5 up, 5 in 2024-03-09T00:00:23.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:23.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:23.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:24.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:23 smithi073 ceph-mon[32309]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:24.199 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:23 smithi179 ceph-mon[36637]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:24.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:23 smithi102 ceph-mon[36106]: from='client.24232 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:25.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:24 smithi073 ceph-mon[32309]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:24 smithi102 ceph-mon[36106]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:24 smithi179 ceph-mon[36637]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:26.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:25 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/3219405087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:25 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:25 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]': finished 2024-03-09T00:00:26.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:25 smithi179 ceph-mon[36637]: osdmap e35: 6 total, 5 up, 6 in 2024-03-09T00:00:26.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:25 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:26.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:25 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/3219405087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:25 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:25 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]': finished 2024-03-09T00:00:26.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:25 smithi073 ceph-mon[32309]: osdmap e35: 6 total, 5 up, 6 in 2024-03-09T00:00:26.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:25 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:26.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:25 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/3219405087' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:25 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]: dispatch 2024-03-09T00:00:26.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:25 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "8b052756-28e2-438c-ad07-0e489a79f2f2"}]': finished 2024-03-09T00:00:26.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:25 smithi102 ceph-mon[36106]: osdmap e35: 6 total, 5 up, 6 in 2024-03-09T00:00:26.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:25 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:27.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:26 smithi073 ceph-mon[32309]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:27.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:26 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/88918327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:27.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:26 smithi102 ceph-mon[36106]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:27.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:26 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/88918327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:26 smithi179 ceph-mon[36637]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:26 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/88918327' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:29.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:28 smithi102 ceph-mon[36106]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:29.309 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:28 smithi179 ceph-mon[36637]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:29.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:28 smithi073 ceph-mon[32309]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:31.243 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:30 smithi179 ceph-mon[36637]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:30 smithi102 ceph-mon[36106]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:30 smithi073 ceph-mon[32309]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:33.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:32 smithi179 ceph-mon[36637]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:33.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-09T00:00:33.212 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:32 smithi102 ceph-mon[36106]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-09T00:00:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:32 smithi073 ceph-mon[32309]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-09T00:00:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:34.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:33 smithi179 ceph-mon[36637]: Deploying daemon osd.5 on smithi179 2024-03-09T00:00:34.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:33 smithi102 ceph-mon[36106]: Deploying daemon osd.5 on smithi179 2024-03-09T00:00:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:33 smithi073 ceph-mon[32309]: Deploying daemon osd.5 on smithi179 2024-03-09T00:00:35.224 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:34 smithi179 ceph-mon[36637]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:35.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:34 smithi102 ceph-mon[36106]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:35.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:34 smithi073 ceph-mon[32309]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:37.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:36 smithi102 ceph-mon[36106]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:37.281 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:36 smithi179 ceph-mon[36637]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:37.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:36 smithi073 ceph-mon[32309]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:38 smithi102 ceph-mon[36106]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:38 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:38 smithi179 ceph-mon[36637]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:38 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:38 smithi073 ceph-mon[32309]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:39.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:39.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:38 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:41.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:40 smithi179 ceph-mon[36637]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:40 smithi102 ceph-mon[36106]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:41.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:40 smithi073 ceph-mon[32309]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:41.680 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.683+0000 7f8a6a7fc700 1 -- 172.21.15.179:0/1879400080 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8a74062220 con 0x7f8a600a2890 2024-03-09T00:00:41.681 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 5 on host 'smithi179' 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 msgr2=0x7f8a600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 0x7f8a600a4d50 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8a64003890 tx=0x7f8a64005fd0 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 msgr2=0x7f8a7419ced0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f8a7000b530 tx=0x7f8a7000b840 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 shutdown_connections 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8a74100d00 0x7f8a7406b0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8a600a2890 0x7f8a600a4d50 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8a741028a0 0x7f8a7419f520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 --2- 172.21.15.179:0/1879400080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8a74101f00 0x7f8a7419ced0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:41.682 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 >> 172.21.15.179:0/1879400080 conn(0x7f8a740fc2e0 msgr2=0x7f8a74105310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:00:41.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 shutdown_connections 2024-03-09T00:00:41.683 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:00:41.686+0000 7f8a7b8f3700 1 -- 172.21.15.179:0/1879400080 wait complete. 2024-03-09T00:00:42.315 DEBUG:teuthology.orchestra.run.smithi179:osd.5> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.5.service 2024-03-09T00:00:42.317 INFO:tasks.cephadm:Deploying osd.6 on smithi073 with /dev/vg_nvme/lv_2... 2024-03-09T00:00:42.317 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-09T00:00:42.562 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:42.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:42 smithi179 ceph-mon[36637]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:42 smithi073 ceph-mon[32309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:43.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:43.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:43.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:43.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:43.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:43.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:43.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:42 smithi102 ceph-mon[36106]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-09T00:00:44.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:43 smithi102 ceph-mon[36106]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-09T00:00:44.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:43 smithi102 ceph-mon[36106]: osdmap e36: 6 total, 5 up, 6 in 2024-03-09T00:00:44.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:43 smithi102 ceph-mon[36106]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:43 smithi102 ceph-mon[36106]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.014 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:43 smithi073 ceph-mon[32309]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-09T00:00:44.014 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:43 smithi073 ceph-mon[32309]: osdmap e36: 6 total, 5 up, 6 in 2024-03-09T00:00:44.014 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.014 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:43 smithi073 ceph-mon[32309]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.014 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:43 smithi073 ceph-mon[32309]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:43 smithi179 ceph-mon[36637]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-09T00:00:44.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:43 smithi179 ceph-mon[36637]: osdmap e36: 6 total, 5 up, 6 in 2024-03-09T00:00:44.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:43 smithi179 ceph-mon[36637]: from='osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:43 smithi179 ceph-mon[36637]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114974 s, 912 MB/s 2024-03-09T00:00:44.680 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-09T00:00:44.969 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: osdmap e37: 6 total, 5 up, 6 in 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: from='osd.5 ' entity='osd.5' 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] boot 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: osdmap e38: 6 total, 6 up, 6 in 2024-03-09T00:00:44.970 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:44.970 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:00:44.612+0000 7f8456ba6700 -1 osd.5 0 waiting for initial osdmap 2024-03-09T00:00:44.970 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:00:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:00:44.620+0000 7f844e190700 -1 osd.5 37 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:00:45.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:00:45.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: osdmap e37: 6 total, 5 up, 6 in 2024-03-09T00:00:45.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:45.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: from='osd.5 ' entity='osd.5' 2024-03-09T00:00:45.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] boot 2024-03-09T00:00:45.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: osdmap e38: 6 total, 6 up, 6 in 2024-03-09T00:00:45.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:44 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.012 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:00:45.012 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: osdmap e37: 6 total, 5 up, 6 in 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 143 MiB used, 447 GiB / 447 GiB avail 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: from='osd.5 ' entity='osd.5' 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: osd.5 [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] boot 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: osdmap e38: 6 total, 6 up, 6 in 2024-03-09T00:00:45.013 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:44 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:00:45.349 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_2 2024-03-09T00:00:45.599 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:45.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: osdmap e39: 6 total, 6 up, 6 in 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:45.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:45 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:00:46.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:00:46.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: osdmap e39: 6 total, 6 up, 6 in 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:46.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:45 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: osdmap e39: 6 total, 6 up, 6 in 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:00:46.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:45 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:00:46.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:46 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi179 2024-03-09T00:00:46.856 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:46 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 8540M 2024-03-09T00:00:46.856 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:46 smithi073 ceph-mon[32309]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:46.856 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:46 smithi073 ceph-mon[32309]: osdmap e40: 6 total, 6 up, 6 in 2024-03-09T00:00:47.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:46 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi179 2024-03-09T00:00:47.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:46 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 8540M 2024-03-09T00:00:47.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:46 smithi102 ceph-mon[36106]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:47.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:46 smithi102 ceph-mon[36106]: osdmap e40: 6 total, 6 up, 6 in 2024-03-09T00:00:47.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:46 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi179 2024-03-09T00:00:47.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:46 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 8540M 2024-03-09T00:00:47.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:46 smithi179 ceph-mon[36637]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:47.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:46 smithi179 ceph-mon[36637]: osdmap e40: 6 total, 6 up, 6 in 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.300+0000 7f5108ed4700 1 -- 172.21.15.73:0/1377836377 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 msgr2=0x7f510410a250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.300+0000 7f5108ed4700 1 --2- 172.21.15.73:0/1377836377 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510410a250 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f50fc00b0a0 tx=0x7f50fc00b3b0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- 172.21.15.73:0/1377836377 shutdown_connections 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- 172.21.15.73:0/1377836377 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510410a250 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- 172.21.15.73:0/1377836377 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 0x7f5104102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- 172.21.15.73:0/1377836377 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 0x7f51041033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- 172.21.15.73:0/1377836377 >> 172.21.15.73:0/1377836377 conn(0x7f51040fd3f0 msgr2=0x7f51040ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:00:47.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- 172.21.15.73:0/1377836377 shutdown_connections 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- 172.21.15.73:0/1377836377 wait complete. 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 Processor -- start 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- start start 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 0x7f5104198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 0x7f5104198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510410da80 con 0x7f5104103930 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510410d900 con 0x7f5104101e10 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.301+0000 7f5108ed4700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510410dc00 con 0x7f5104102f70 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:47.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38782/0 (socket says 172.21.15.73:38782) 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 -- 172.21.15.73:0/137249175 learned_addr learned my addr 172.21.15.73:0/137249175 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f510259c700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 0x7f5104198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5101d9b700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 0x7f5104198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 -- 172.21.15.73:0/137249175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 msgr2=0x7f5104198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 0x7f5104198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 -- 172.21.15.73:0/137249175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 msgr2=0x7f5104198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 0x7f5104198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 -- 172.21.15.73:0/137249175 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f510419fc60 con 0x7f5104103930 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5102d9d700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007d90 tx=0x7f50fc007dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc01f070 con 0x7f5104103930 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50fc003e10 con 0x7f5104103930 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f510419fef0 con 0x7f5104103930 2024-03-09T00:00:47.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50fc01bab0 con 0x7f5104103930 2024-03-09T00:00:47.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.302+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51041a0360 con 0x7f5104103930 2024-03-09T00:00:47.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.303+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5104068aa0 con 0x7f5104103930 2024-03-09T00:00:47.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.304+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f50fc022070 con 0x7f5104103930 2024-03-09T00:00:47.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.305+0000 7f50f37fe700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 0x7f50ec0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:00:47.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.305+0000 7f510259c700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 0x7f50ec0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:00:47.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.305+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4562+0+0 (secure 0 0 0) 0x7f50fc0cee90 con 0x7f5104103930 2024-03-09T00:00:47.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.306+0000 7f510259c700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 0x7f50ec0a4d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009a10 tx=0x7f50f4009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:00:47.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.310+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f50fc029020 con 0x7f5104103930 2024-03-09T00:00:47.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:00:47.513+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5104062220 con 0x7f50ec0a2890 2024-03-09T00:00:48.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:47 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:48.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:47 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:48.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:47 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:48.107 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:47 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:48.107 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:47 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:48.107 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:47 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:48.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:47 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:00:48.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:47 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:00:48.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:47 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:49.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:48 smithi179 ceph-mon[36637]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:49.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:48 smithi179 ceph-mon[36637]: osdmap e41: 6 total, 6 up, 6 in 2024-03-09T00:00:49.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:48 smithi179 ceph-mon[36637]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:00:49.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:48 smithi179 ceph-mon[36637]: osdmap e42: 6 total, 6 up, 6 in 2024-03-09T00:00:49.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:48 smithi073 ceph-mon[32309]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:48 smithi073 ceph-mon[32309]: osdmap e41: 6 total, 6 up, 6 in 2024-03-09T00:00:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:48 smithi073 ceph-mon[32309]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:00:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:48 smithi073 ceph-mon[32309]: osdmap e42: 6 total, 6 up, 6 in 2024-03-09T00:00:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:48 smithi102 ceph-mon[36106]: from='client.14361 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:00:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:48 smithi102 ceph-mon[36106]: osdmap e41: 6 total, 6 up, 6 in 2024-03-09T00:00:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:48 smithi102 ceph-mon[36106]: pgmap v143: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 570 MiB used, 536 GiB / 536 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:00:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:48 smithi102 ceph-mon[36106]: osdmap e42: 6 total, 6 up, 6 in 2024-03-09T00:00:51.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:50 smithi073 ceph-mon[32309]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:51.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:50 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]: dispatch 2024-03-09T00:00:51.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:50 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]': finished 2024-03-09T00:00:51.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:50 smithi073 ceph-mon[32309]: osdmap e43: 7 total, 6 up, 7 in 2024-03-09T00:00:51.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:00:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:50 smithi102 ceph-mon[36106]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:50 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]: dispatch 2024-03-09T00:00:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:50 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]': finished 2024-03-09T00:00:51.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:50 smithi102 ceph-mon[36106]: osdmap e43: 7 total, 6 up, 7 in 2024-03-09T00:00:51.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:00:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:50 smithi179 ceph-mon[36637]: pgmap v145: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-09T00:00:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:50 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]: dispatch 2024-03-09T00:00:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:50 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/450688954' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "4be6204e-f4a1-4605-a89f-3b3e222336c1"}]': finished 2024-03-09T00:00:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:50 smithi179 ceph-mon[36637]: osdmap e43: 7 total, 6 up, 7 in 2024-03-09T00:00:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:00:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:51 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1357884480' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:52.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:51 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1357884480' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:51 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1357884480' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:00:53.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:52 smithi102 ceph-mon[36106]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:53.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:52 smithi179 ceph-mon[36637]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:53.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:52 smithi073 ceph-mon[32309]: pgmap v147: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 178 MiB used, 536 GiB / 536 GiB avail; 111 KiB/s, 0 objects/s recovering 2024-03-09T00:00:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:54 smithi102 ceph-mon[36106]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:54 smithi179 ceph-mon[36637]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:54 smithi073 ceph-mon[32309]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:57.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:56 smithi073 ceph-mon[32309]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:57.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:56 smithi102 ceph-mon[36106]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:57.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:56 smithi179 ceph-mon[36637]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:58.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:57 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-09T00:00:58.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:57 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:58.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:57 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-09T00:00:58.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:57 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:57 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-09T00:00:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:57 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:00:59.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:58 smithi073 ceph-mon[32309]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:59.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:00:58 smithi073 ceph-mon[32309]: Deploying daemon osd.6 on smithi073 2024-03-09T00:00:59.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:58 smithi102 ceph-mon[36106]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:59.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:00:58 smithi102 ceph-mon[36106]: Deploying daemon osd.6 on smithi073 2024-03-09T00:00:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:58 smithi179 ceph-mon[36637]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:00:59.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:00:58 smithi179 ceph-mon[36637]: Deploying daemon osd.6 on smithi073 2024-03-09T00:01:01.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:00 smithi073 ceph-mon[32309]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:01.220 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:00 smithi179 ceph-mon[36637]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:01.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:00 smithi102 ceph-mon[36106]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:03.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:02 smithi102 ceph-mon[36106]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:03.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:02 smithi179 ceph-mon[36637]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:03.357 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:02 smithi073 ceph-mon[32309]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:05.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:04 smithi073 ceph-mon[32309]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:05.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:05.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:05.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:05.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:04 smithi102 ceph-mon[36106]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:05.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:05.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:05.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:04 smithi179 ceph-mon[36637]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:07.248 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:06 smithi073 ceph-mon[32309]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:07.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:06 smithi102 ceph-mon[36106]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:06 smithi179 ceph-mon[36637]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:07.746 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 6 on host 'smithi073' 2024-03-09T00:01:07.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.750+0000 7f50f37fe700 1 -- 172.21.15.73:0/137249175 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5104062220 con 0x7f50ec0a2890 2024-03-09T00:01:07.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 msgr2=0x7f50ec0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:07.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 0x7f50ec0a4d50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f50f4009a10 tx=0x7f50f4009190 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 msgr2=0x7f510419f5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f50fc007d90 tx=0x7f50fc007dc0 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 shutdown_connections 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5104103930 0x7f510419f5c0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50ec0a2890 0x7f50ec0a4d50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5104101e10 0x7f5104198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 --2- 172.21.15.73:0/137249175 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5104102f70 0x7f5104198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 >> 172.21.15.73:0/137249175 conn(0x7f51040fd3f0 msgr2=0x7f51041062a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 shutdown_connections 2024-03-09T00:01:07.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:01:07.752+0000 7f5108ed4700 1 -- 172.21.15.73:0/137249175 wait complete. 2024-03-09T00:01:08.495 DEBUG:teuthology.orchestra.run.smithi073:osd.6> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.6.service 2024-03-09T00:01:08.497 INFO:tasks.cephadm:Deploying osd.7 on smithi102 with /dev/vg_nvme/lv_2... 2024-03-09T00:01:08.497 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-09T00:01:08.736 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.858 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-09T00:01:08.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:08.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:08.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:09.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:09.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:09 smithi073 ceph-mon[32309]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-09T00:01:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:09 smithi073 ceph-mon[32309]: osdmap e44: 7 total, 6 up, 7 in 2024-03-09T00:01:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:09 smithi073 ceph-mon[32309]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:01:09.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:10.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:09 smithi102 ceph-mon[36106]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-09T00:01:10.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:09 smithi102 ceph-mon[36106]: osdmap e44: 7 total, 6 up, 7 in 2024-03-09T00:01:10.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:09 smithi102 ceph-mon[36106]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:01:10.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:10.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:09 smithi179 ceph-mon[36637]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-09T00:01:10.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:09 smithi179 ceph-mon[36637]: osdmap e44: 7 total, 6 up, 7 in 2024-03-09T00:01:10.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:09 smithi179 ceph-mon[36637]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:01:10.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:10.652 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:01:10.459+0000 7fd222435700 -1 osd.6 0 waiting for initial osdmap 2024-03-09T00:01:10.652 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:01:10.468+0000 7fd21a220700 -1 osd.6 45 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113867 s, 921 MB/s 2024-03-09T00:01:10.926 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-09T00:01:11.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:01:11.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: osdmap e45: 7 total, 6 up, 7 in 2024-03-09T00:01:11.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' 2024-03-09T00:01:11.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:10 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: osdmap e45: 7 total, 6 up, 7 in 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' 2024-03-09T00:01:11.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:10 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: osdmap e45: 7 total, 6 up, 7 in 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 179 MiB used, 536 GiB / 536 GiB avail 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: from='osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006]' entity='osd.6' 2024-03-09T00:01:11.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:10 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:11.608 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_2 2024-03-09T00:01:11.854 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:01:11.875 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:11 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:01:11.876 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:11 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:01:11.876 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:11 smithi102 ceph-mon[36106]: osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] boot 2024-03-09T00:01:11.876 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:11 smithi102 ceph-mon[36106]: osdmap e46: 7 total, 7 up, 7 in 2024-03-09T00:01:11.876 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:12.004 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:11 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:01:12.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:11 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:01:12.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:11 smithi073 ceph-mon[32309]: osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] boot 2024-03-09T00:01:12.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:11 smithi073 ceph-mon[32309]: osdmap e46: 7 total, 7 up, 7 in 2024-03-09T00:01:12.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:12.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:11 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:01:12.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:11 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:01:12.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:11 smithi179 ceph-mon[36637]: osd.6 [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] boot 2024-03-09T00:01:12.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:11 smithi179 ceph-mon[36637]: osdmap e46: 7 total, 7 up, 7 in 2024-03-09T00:01:12.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:12.979 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:12.980 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:12.980 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:12.980 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:13.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:13.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 209 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:13.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 -- 172.21.15.102:0/4289917718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 msgr2=0x7f4f341054c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/4289917718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f341054c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c009990 tx=0x7f4f1c009ca0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 -- 172.21.15.102:0/4289917718 shutdown_connections 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/4289917718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f34105a00 0x7f4f34105e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/4289917718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f341006c0 0x7f4f34102af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/4289917718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f341054c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.494 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 -- 172.21.15.102:0/4289917718 >> 172.21.15.102:0/4289917718 conn(0x7f4f340fc2e0 msgr2=0x7f4f340fe720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:01:13.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 -- 172.21.15.102:0/4289917718 shutdown_connections 2024-03-09T00:01:13.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.498+0000 7f4f39c9b700 1 -- 172.21.15.102:0/4289917718 wait complete. 2024-03-09T00:01:13.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 Processor -- start 2024-03-09T00:01:13.495 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 -- start start 2024-03-09T00:01:13.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f341006c0 0x7f4f3419e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:13.496 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 0x7f4f341a5b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f34113fa0 con 0x7f4f341006c0 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f34113e20 con 0x7f4f34103030 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f39c9b700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f34114120 con 0x7f4f34105a00 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f337fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f341006c0 0x7f4f3419e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:58606/0 (socket says 172.21.15.102:58606) 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 -- 172.21.15.102:0/2502874721 learned_addr learned my addr 172.21.15.102:0/2502874721 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f33fff700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 0x7f4f341a5b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 -- 172.21.15.102:0/2502874721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 msgr2=0x7f4f341a5b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 0x7f4f341a5b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 -- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f341006c0 msgr2=0x7f4f3419e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:13.497 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f341006c0 0x7f4f3419e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 -- 172.21.15.102:0/2502874721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f341a61b0 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f33fff700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 0x7f4f341a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.499+0000 7f4f2bfff700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0098e0 tx=0x7f4f1c00f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.500+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c027070 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.500+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f341a6440 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.500+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f1c022950 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.500+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f1c017750 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.500+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f341a69a0 con 0x7f4f34103030 2024-03-09T00:01:13.498 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.501+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f34068aa0 con 0x7f4f34103030 2024-03-09T00:01:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.502+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f1c02a070 con 0x7f4f34103030 2024-03-09T00:01:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.502+0000 7f4f317fa700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 0x7f4f200a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.502+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f4f1c0cef10 con 0x7f4f34103030 2024-03-09T00:01:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.503+0000 7f4f337fe700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 0x7f4f200a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:13.499 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.503+0000 7f4f337fe700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 0x7f4f200a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f24004630 tx=0x7f4f2400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:01:13.501 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.506+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4f1c013070 con 0x7f4f34103030 2024-03-09T00:01:13.706 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:13.710+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4f340622e0 con 0x7f4f200a2890 2024-03-09T00:01:14.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi073 2024-03-09T00:01:14.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 5696M 2024-03-09T00:01:14.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: osdmap e47: 7 total, 7 up, 7 in 2024-03-09T00:01:14.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:14.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi073 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 5696M 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: osdmap e47: 7 total, 7 up, 7 in 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:14.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:14.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi073 2024-03-09T00:01:14.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 5696M 2024-03-09T00:01:14.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: osdmap e47: 7 total, 7 up, 7 in 2024-03-09T00:01:14.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:14.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:14.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:14 smithi102 ceph-mon[36106]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:14 smithi102 ceph-mon[36106]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:15.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:14 smithi179 ceph-mon[36637]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:15.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:14 smithi179 ceph-mon[36637]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:15.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:14 smithi073 ceph-mon[32309]: from='client.24260 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:15.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:14 smithi073 ceph-mon[32309]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:17.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:17.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/1757562741' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.185 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.186 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]': finished 2024-03-09T00:01:17.186 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: osdmap e48: 8 total, 7 up, 8 in 2024-03-09T00:01:17.186 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/1757562741' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]': finished 2024-03-09T00:01:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: osdmap e48: 8 total, 7 up, 8 in 2024-03-09T00:01:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/1757562741' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]: dispatch 2024-03-09T00:01:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e08563fd-4504-49ad-9b28-33db12548e5e"}]': finished 2024-03-09T00:01:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: osdmap e48: 8 total, 7 up, 8 in 2024-03-09T00:01:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:17 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/1721559365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:18.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:17 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/1721559365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:17 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/1721559365' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:18 smithi102 ceph-mon[36106]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:19.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:18 smithi179 ceph-mon[36637]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:19.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:18 smithi073 ceph-mon[32309]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:21.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:20 smithi102 ceph-mon[36106]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:21.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:20 smithi179 ceph-mon[36637]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:21.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:20 smithi073 ceph-mon[32309]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:23.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:22 smithi102 ceph-mon[36106]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:23.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:22 smithi179 ceph-mon[36637]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:23.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:22 smithi073 ceph-mon[32309]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:24.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:23 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-09T00:01:24.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:23 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:24.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:23 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-09T00:01:24.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:23 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:24.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:23 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-09T00:01:24.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:23 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:24 smithi102 ceph-mon[36106]: Deploying daemon osd.7 on smithi102 2024-03-09T00:01:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:24 smithi102 ceph-mon[36106]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:24 smithi179 ceph-mon[36637]: Deploying daemon osd.7 on smithi102 2024-03-09T00:01:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:24 smithi179 ceph-mon[36637]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:25.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:24 smithi073 ceph-mon[32309]: Deploying daemon osd.7 on smithi102 2024-03-09T00:01:25.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:24 smithi073 ceph-mon[32309]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:27.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:26 smithi102 ceph-mon[36106]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:27.204 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:26 smithi073 ceph-mon[32309]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:27.245 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:26 smithi179 ceph-mon[36637]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:29.189 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:28 smithi102 ceph-mon[36106]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:28 smithi179 ceph-mon[36637]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:29.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:28 smithi073 ceph-mon[32309]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:30.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:30.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:30 smithi102 ceph-mon[36106]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:30 smithi179 ceph-mon[36637]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:30.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:30 smithi073 ceph-mon[32309]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:33.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:33.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.157 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.158 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:33.158 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:33.158 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:32 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.221 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 7 on host 'smithi102' 2024-03-09T00:01:33.222 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.226+0000 7f4f317fa700 1 -- 172.21.15.102:0/2502874721 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4f340622e0 con 0x7f4f200a2890 2024-03-09T00:01:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 msgr2=0x7f4f200a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 0x7f4f200a4d50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f4f24004630 tx=0x7f4f2400b3c0 comp rx=0 tx=0).stop 2024-03-09T00:01:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 msgr2=0x7f4f3419ea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4f1c0098e0 tx=0x7f4f1c00f690 comp rx=0 tx=0).stop 2024-03-09T00:01:33.224 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 shutdown_connections 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f341006c0 0x7f4f3419e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f200a2890 0x7f4f200a4d50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f34105a00 0x7f4f341a5b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 --2- 172.21.15.102:0/2502874721 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f34103030 0x7f4f3419ea90 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 >> 172.21.15.102:0/2502874721 conn(0x7f4f340fc2e0 msgr2=0x7f4f34101370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 shutdown_connections 2024-03-09T00:01:33.225 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:01:33.229+0000 7f4f39c9b700 1 -- 172.21.15.102:0/2502874721 wait complete. 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:33.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:32 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:32 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:33.929 DEBUG:teuthology.orchestra.run.smithi102:osd.7> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.7.service 2024-03-09T00:01:33.932 INFO:tasks.cephadm:Deploying osd.8 on smithi179 with /dev/vg_nvme/lv_2... 2024-03-09T00:01:33.933 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-09T00:01:34.180 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:34 smithi179 ceph-mon[36637]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:34 smithi102 ceph-mon[36106]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:34.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-09T00:01:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:34 smithi073 ceph-mon[32309]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:35.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:35 smithi179 ceph-mon[36637]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-09T00:01:35.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:35 smithi179 ceph-mon[36637]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:35.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:35 smithi179 ceph-mon[36637]: osdmap e49: 8 total, 7 up, 8 in 2024-03-09T00:01:35.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:35 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:35.249 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:35 smithi179 ceph-mon[36637]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:35.472 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-mon[36106]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-09T00:01:35.472 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-mon[36106]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:35.472 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-mon[36106]: osdmap e49: 8 total, 7 up, 8 in 2024-03-09T00:01:35.472 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:35.472 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-mon[36106]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:35.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:35 smithi073 ceph-mon[32309]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-09T00:01:35.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:35 smithi073 ceph-mon[32309]: from='osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:35.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:35 smithi073 ceph-mon[32309]: osdmap e49: 8 total, 7 up, 8 in 2024-03-09T00:01:35.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:35 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:35.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:35 smithi073 ceph-mon[32309]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:01:36.058 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-09T00:01:36.059 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-09T00:01:36.059 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-09T00:01:36.059 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-09T00:01:36.059 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.013379 s, 784 MB/s 2024-03-09T00:01:36.059 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-09T00:01:36.215 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:01:35.886+0000 7f5e8eb64700 -1 osd.7 0 waiting for initial osdmap 2024-03-09T00:01:36.215 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:01:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:01:35.894+0000 7f5e8694f700 -1 osd.7 50 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: osdmap e50: 8 total, 7 up, 8 in 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:36.358 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:36 smithi179 ceph-mon[36637]: from='osd.7 ' entity='osd.7' 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: osdmap e50: 8 total, 7 up, 8 in 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:36 smithi102 ceph-mon[36106]: from='osd.7 ' entity='osd.7' 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: osdmap e50: 8 total, 7 up, 8 in 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 210 MiB used, 626 GiB / 626 GiB avail 2024-03-09T00:01:36.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:36 smithi073 ceph-mon[32309]: from='osd.7 ' entity='osd.7' 2024-03-09T00:01:36.721 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_2 2024-03-09T00:01:36.967 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] boot 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: osdmap e51: 8 total, 8 up, 8 in 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi102 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.347 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 5693M 2024-03-09T00:01:37.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:37.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:37.348 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] boot 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: osdmap e51: 8 total, 8 up, 8 in 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi102 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 5693M 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:37.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:37.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: osd.7 [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] boot 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: osdmap e51: 8 total, 8 up, 8 in 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi102 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 5693M 2024-03-09T00:01:37.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:37.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:37.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:37.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:38.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:38 smithi179 ceph-mon[36637]: osdmap e52: 8 total, 8 up, 8 in 2024-03-09T00:01:38.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:38 smithi179 ceph-mon[36637]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:38.494 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:38 smithi073 ceph-mon[32309]: osdmap e52: 8 total, 8 up, 8 in 2024-03-09T00:01:38.494 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:38 smithi073 ceph-mon[32309]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:38 smithi102 ceph-mon[36106]: osdmap e52: 8 total, 8 up, 8 in 2024-03-09T00:01:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:38 smithi102 ceph-mon[36106]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- 172.21.15.179:0/639094967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 msgr2=0x7f6ebc109fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 --2- 172.21.15.179:0/639094967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc109fb0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400b0a0 tx=0x7f6eb400b3b0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- 172.21.15.179:0/639094967 shutdown_connections 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 --2- 172.21.15.179:0/639094967 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc109fb0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 --2- 172.21.15.179:0/639094967 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc102960 0x7f6ebc104d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 --2- 172.21.15.179:0/639094967 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc105290 0x7f6ebc107680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.529 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- 172.21.15.179:0/639094967 >> 172.21.15.179:0/639094967 conn(0x7f6ebc0fc2d0 msgr2=0x7f6ebc0fe730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- 172.21.15.179:0/639094967 shutdown_connections 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- 172.21.15.179:0/639094967 wait complete. 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 Processor -- start 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 -- start start 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.529+0000 7f6ec41da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 0x7f6ebc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc105290 0x7f6ebc1984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc10da00 con 0x7f6ebc102960 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc10d880 con 0x7f6ebc105290 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ebc10db80 con 0x7f6ebc107bc0 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec1f76700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 0x7f6ebc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:52830/0 (socket says 172.21.15.179:52830) 2024-03-09T00:01:38.530 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 -- 172.21.15.179:0/3549385094 learned_addr learned my addr 172.21.15.179:0/3549385094 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec1775700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc105290 0x7f6ebc1984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 -- 172.21.15.179:0/3549385094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc105290 msgr2=0x7f6ebc1984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc105290 0x7f6ebc1984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 -- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 msgr2=0x7f6ebc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 0x7f6ebc197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 -- 172.21.15.179:0/3549385094 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ebc19fbc0 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec1f76700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 0x7f6ebc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec2777700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400b070 tx=0x7f6eb4004d00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb4021070 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6eb4007b80 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ebc19fe50 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6eb401d450 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.530+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ebc1a02a0 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.531+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ebc068aa0 con 0x7f6ebc107bc0 2024-03-09T00:01:38.531 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.532+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6eb4003a00 con 0x7f6ebc107bc0 2024-03-09T00:01:38.532 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.533+0000 7f6eb2ffd700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 0x7f6ea80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:01:38.532 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.533+0000 7f6ec1f76700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 0x7f6ea80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:01:38.532 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.533+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 5426+0+0 (secure 0 0 0) 0x7f6eb400f020 con 0x7f6ebc107bc0 2024-03-09T00:01:38.532 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.534+0000 7f6ec1f76700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 0x7f6ea80a4d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8010420 tx=0x7f6eb80073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:01:38.532 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.538+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6eb402a020 con 0x7f6ebc107bc0 2024-03-09T00:01:38.738 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:38.743+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6ebc062220 con 0x7f6ea80a2890 2024-03-09T00:01:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:39 smithi102 ceph-mon[36106]: osdmap e53: 8 total, 8 up, 8 in 2024-03-09T00:01:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:39 smithi102 ceph-mon[36106]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:39 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:39 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:39 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:39.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:39 smithi179 ceph-mon[36637]: osdmap e53: 8 total, 8 up, 8 in 2024-03-09T00:01:39.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:39 smithi179 ceph-mon[36637]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:39.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:39 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:39.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:39 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:39.541 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:39 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:39.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:39 smithi073 ceph-mon[32309]: osdmap e53: 8 total, 8 up, 8 in 2024-03-09T00:01:39.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:39 smithi073 ceph-mon[32309]: from='client.24292 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:01:39.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:39 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:01:39.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:39 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:01:39.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:39 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:40 smithi102 ceph-mon[36106]: osdmap e54: 8 total, 8 up, 8 in 2024-03-09T00:01:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:40 smithi102 ceph-mon[36106]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:40 smithi102 ceph-mon[36106]: osdmap e55: 8 total, 8 up, 8 in 2024-03-09T00:01:40.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:40 smithi179 ceph-mon[36637]: osdmap e54: 8 total, 8 up, 8 in 2024-03-09T00:01:40.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:40 smithi179 ceph-mon[36637]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:40.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:40 smithi179 ceph-mon[36637]: osdmap e55: 8 total, 8 up, 8 in 2024-03-09T00:01:40.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:40 smithi073 ceph-mon[32309]: osdmap e54: 8 total, 8 up, 8 in 2024-03-09T00:01:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:40 smithi073 ceph-mon[32309]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 492 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:40.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:40 smithi073 ceph-mon[32309]: osdmap e55: 8 total, 8 up, 8 in 2024-03-09T00:01:41.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:41 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/2490672048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:41 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:41 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]': finished 2024-03-09T00:01:41.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:41 smithi102 ceph-mon[36106]: osdmap e56: 9 total, 8 up, 9 in 2024-03-09T00:01:41.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:41 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:01:41.801 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:41 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/2490672048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:41 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:41 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]': finished 2024-03-09T00:01:41.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:41 smithi179 ceph-mon[36637]: osdmap e56: 9 total, 8 up, 9 in 2024-03-09T00:01:41.802 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:41 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:01:41.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:41 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/2490672048' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:41 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]: dispatch 2024-03-09T00:01:41.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:41 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b60fba7-28c7-455e-9120-944eaa7f08dd"}]': finished 2024-03-09T00:01:41.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:41 smithi073 ceph-mon[32309]: osdmap e56: 9 total, 8 up, 9 in 2024-03-09T00:01:41.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:41 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:01:42.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:42 smithi102 ceph-mon[36106]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:42.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:42 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/229574180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:42 smithi179 ceph-mon[36637]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:42.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:42 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/229574180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:42.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:42 smithi073 ceph-mon[32309]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 242 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:42.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:42 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/229574180' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:01:45.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:44 smithi102 ceph-mon[36106]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:45.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:44 smithi179 ceph-mon[36637]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:44 smithi073 ceph-mon[32309]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:47.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:46 smithi179 ceph-mon[36637]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:46 smithi102 ceph-mon[36106]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:47.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:46 smithi073 ceph-mon[32309]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:48 smithi102 ceph-mon[36106]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-09T00:01:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:48 smithi179 ceph-mon[36637]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-09T00:01:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:48 smithi073 ceph-mon[32309]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-09T00:01:49.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:50.183 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:49 smithi179 ceph-mon[36637]: Deploying daemon osd.8 on smithi179 2024-03-09T00:01:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:49 smithi102 ceph-mon[36106]: Deploying daemon osd.8 on smithi179 2024-03-09T00:01:50.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:49 smithi073 ceph-mon[32309]: Deploying daemon osd.8 on smithi179 2024-03-09T00:01:51.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:50 smithi179 ceph-mon[36637]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:50 smithi102 ceph-mon[36106]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:51.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:50 smithi073 ceph-mon[32309]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:52 smithi179 ceph-mon[36637]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:53.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:52 smithi102 ceph-mon[36106]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:52 smithi073 ceph-mon[32309]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:54 smithi102 ceph-mon[36106]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:55.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:54 smithi179 ceph-mon[36637]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:55.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.276 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:54 smithi073 ceph-mon[32309]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:57.203 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:56 smithi073 ceph-mon[32309]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:57.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:56 smithi102 ceph-mon[36106]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:57.291 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:56 smithi179 ceph-mon[36637]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:58.454 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 8 on host 'smithi179' 2024-03-09T00:01:58.454 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.458+0000 7f6eb2ffd700 1 -- 172.21.15.179:0/3549385094 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6ebc062220 con 0x7f6ea80a2890 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 msgr2=0x7f6ea80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 0x7f6ea80a4d50 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f6eb8010420 tx=0x7f6eb80073d0 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 msgr2=0x7f6ebc19f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f6eb400b070 tx=0x7f6eb4004d00 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 shutdown_connections 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ebc102960 0x7f6ebc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6ea80a2890 0x7f6ea80a4d50 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ebc107bc0 0x7f6ebc19f520 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 --2- 172.21.15.179:0/3549385094 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ebc105290 0x7f6ebc1984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:01:58.457 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 >> 172.21.15.179:0/3549385094 conn(0x7f6ebc0fc2d0 msgr2=0x7f6ebc1032a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:01:58.458 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 shutdown_connections 2024-03-09T00:01:58.458 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:01:58.461+0000 7f6ec41da700 1 -- 172.21.15.179:0/3549385094 wait complete. 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.690 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:58.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:58.691 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:01:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.854 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:58.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:58.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:58.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:58.855 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:01:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:59.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:59.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:59.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:01:59.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:01:59.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-09T00:01:59.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:01:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:01:59.086 DEBUG:teuthology.orchestra.run.smithi179:osd.8> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.8.service 2024-03-09T00:01:59.089 INFO:tasks.cephadm:Deploying osd.9 on smithi073 with /dev/vg_nvme/lv_1... 2024-03-09T00:01:59.089 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-09T00:01:59.321 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: osdmap e57: 9 total, 8 up, 9 in 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.560 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-mon[36637]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:00.560 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:02:00.447+0000 7f55aa380700 -1 osd.8 0 waiting for initial osdmap 2024-03-09T00:02:00.561 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:02:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:02:00.456+0000 7f55a6173700 -1 osd.8 58 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: osdmap e57: 9 total, 8 up, 9 in 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:00 smithi102 ceph-mon[36106]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:00.764 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-09T00:02:00.764 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: osdmap e57: 9 total, 8 up, 9 in 2024-03-09T00:02:00.765 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: from='osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.765 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:00.765 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:02:00.765 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:00 smithi073 ceph-mon[32309]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:01.480 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-09T00:02:01.480 INFO:teuthology.orchestra.run.smithi073.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-09T00:02:01.480 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10+0 records in 2024-03-09T00:02:01.480 INFO:teuthology.orchestra.run.smithi073.stderr:10+0 records out 2024-03-09T00:02:01.480 INFO:teuthology.orchestra.run.smithi073.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114191 s, 918 MB/s 2024-03-09T00:02:01.481 INFO:teuthology.orchestra.run.smithi073.stderr:--> Zapping successful for: 2024-03-09T00:02:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:01 smithi102 ceph-mon[36106]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:02:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:01 smithi102 ceph-mon[36106]: osdmap e58: 9 total, 8 up, 9 in 2024-03-09T00:02:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:01 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:01 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:01.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:01 smithi179 ceph-mon[36637]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:02:01.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:01 smithi179 ceph-mon[36637]: osdmap e58: 9 total, 8 up, 9 in 2024-03-09T00:02:01.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:01 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:01.800 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:01 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:01.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:01 smithi073 ceph-mon[32309]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:02:01.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:01 smithi073 ceph-mon[32309]: osdmap e58: 9 total, 8 up, 9 in 2024-03-09T00:02:01.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:01 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:01.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:01 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.132 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi073:vg_nvme/lv_1 2024-03-09T00:02:02.370 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] boot 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: osdmap e59: 9 total, 9 up, 9 in 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:02 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.750 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:02:02.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:02:02.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] boot 2024-03-09T00:02:02.979 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: osdmap e59: 9 total, 9 up, 9 in 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:02.980 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:02 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: osd.8 [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] boot 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: osdmap e59: 9 total, 9 up, 9 in 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: pgmap v198: 1 pgs: 1 peering; 577 KiB data, 241 MiB used, 715 GiB / 715 GiB avail 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.981 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:02.982 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:02 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:03.521 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:03 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi179 2024-03-09T00:02:03.521 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:03 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 5693M 2024-03-09T00:02:03.521 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:03 smithi073 ceph-mon[32309]: osdmap e60: 9 total, 9 up, 9 in 2024-03-09T00:02:03.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:03 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi179 2024-03-09T00:02:03.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:03 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 5693M 2024-03-09T00:02:03.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:03 smithi102 ceph-mon[36106]: osdmap e60: 9 total, 9 up, 9 in 2024-03-09T00:02:03.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:03 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi179 2024-03-09T00:02:03.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:03 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 5693M 2024-03-09T00:02:03.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:03 smithi179 ceph-mon[36637]: osdmap e60: 9 total, 9 up, 9 in 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 -- 172.21.15.73:0/3893180603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 msgr2=0x7f8fa8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 --2- 172.21.15.73:0/3893180603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8101130 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c0099c0 tx=0x7f8f9c009cd0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 -- 172.21.15.73:0/3893180603 shutdown_connections 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 --2- 172.21.15.73:0/3893180603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8101130 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 --2- 172.21.15.73:0/3893180603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa8109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.072+0000 7f8faf22e700 1 --2- 172.21.15.73:0/3893180603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fa8101f10 0x7f8fa8102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- 172.21.15.73:0/3893180603 >> 172.21.15.73:0/3893180603 conn(0x7f8fa80fc2d0 msgr2=0x7f8fa80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- 172.21.15.73:0/3893180603 shutdown_connections 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- 172.21.15.73:0/3893180603 wait complete. 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 Processor -- start 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- start start 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fa8101f10 0x7f8fa8197440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa819e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa810c940 con 0x7f8fa8100d10 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa810c7c0 con 0x7f8fa81028b0 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8faf22e700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa810cac0 con 0x7f8fa8101f10 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8facfca700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8facfca700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47044/0 (socket says 172.21.15.73:47044) 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.073+0000 7f8facfca700 1 -- 172.21.15.73:0/820463521 learned_addr learned my addr 172.21.15.73:0/820463521 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:02:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8fad7cb700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa819e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 -- 172.21.15.73:0/820463521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fa8101f10 msgr2=0x7f8fa8197440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fa8101f10 0x7f8fa8197440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 -- 172.21.15.73:0/820463521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 msgr2=0x7f8fa819e4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa819e4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 -- 172.21.15.73:0/820463521 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fa819eb60 con 0x7f8fa8100d10 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8fad7cb700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa819e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8facfca700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00c010 tx=0x7f8f9c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c027070 con 0x7f8fa8100d10 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f9c022470 con 0x7f8fa8100d10 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa819edf0 con 0x7f8fa8100d10 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f9c00f6a0 con 0x7f8fa8100d10 2024-03-09T00:02:04.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.074+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa806a720 con 0x7f8fa8100d10 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.075+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fa8068aa0 con 0x7f8fa8100d10 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.076+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f9c02a070 con 0x7f8fa8100d10 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.077+0000 7f8fa5ffb700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 0x7f8f980a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.077+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 5858+0+0 (secure 0 0 0) 0x7f8f9c0cf480 con 0x7f8fa8100d10 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.077+0000 7f8fa7fff700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 0x7f8f980a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:04.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.077+0000 7f8fa7fff700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 0x7f8f980a4dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400a990 tx=0x7f8f94005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:04.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.081+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8f9c013070 con 0x7f8fa8100d10 2024-03-09T00:02:04.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:04.288+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f8fa8107e90 con 0x7f8f980a2910 2024-03-09T00:02:04.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:04 smithi073 ceph-mon[32309]: osdmap e61: 9 total, 9 up, 9 in 2024-03-09T00:02:04.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:04 smithi073 ceph-mon[32309]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:04.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:04.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:04.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:04.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:04 smithi102 ceph-mon[36106]: osdmap e61: 9 total, 9 up, 9 in 2024-03-09T00:02:04.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:04 smithi102 ceph-mon[36106]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:04.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:04.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:04.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:04.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:04 smithi179 ceph-mon[36637]: osdmap e61: 9 total, 9 up, 9 in 2024-03-09T00:02:04.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:04 smithi179 ceph-mon[36637]: pgmap v201: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:04.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:04.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:04.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:05.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:05 smithi073 ceph-mon[32309]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:05.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:05 smithi073 ceph-mon[32309]: osdmap e62: 9 total, 9 up, 9 in 2024-03-09T00:02:05.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:05 smithi102 ceph-mon[36106]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:05.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:05 smithi102 ceph-mon[36106]: osdmap e62: 9 total, 9 up, 9 in 2024-03-09T00:02:05.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:05 smithi179 ceph-mon[36637]: from='client.14427 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi073:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:05.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:05 smithi179 ceph-mon[36637]: osdmap e62: 9 total, 9 up, 9 in 2024-03-09T00:02:06.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:06 smithi102 ceph-mon[36106]: osdmap e63: 9 total, 9 up, 9 in 2024-03-09T00:02:06.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:06 smithi102 ceph-mon[36106]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:06.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:06 smithi179 ceph-mon[36637]: osdmap e63: 9 total, 9 up, 9 in 2024-03-09T00:02:06.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:06 smithi179 ceph-mon[36637]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:06.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:06 smithi073 ceph-mon[32309]: osdmap e63: 9 total, 9 up, 9 in 2024-03-09T00:02:06.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:06 smithi073 ceph-mon[32309]: pgmap v204: 1 pgs: 1 peering; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:07.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]: dispatch 2024-03-09T00:02:07.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]': finished 2024-03-09T00:02:07.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:07 smithi102 ceph-mon[36106]: osdmap e64: 10 total, 9 up, 10 in 2024-03-09T00:02:07.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:07 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:07.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]: dispatch 2024-03-09T00:02:07.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]': finished 2024-03-09T00:02:07.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:07 smithi179 ceph-mon[36637]: osdmap e64: 10 total, 9 up, 10 in 2024-03-09T00:02:07.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:07 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:07.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]: dispatch 2024-03-09T00:02:07.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2688129276' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f6075430-0ebe-4194-9acd-b33cbed4bdd1"}]': finished 2024-03-09T00:02:07.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:07 smithi073 ceph-mon[32309]: osdmap e64: 10 total, 9 up, 10 in 2024-03-09T00:02:07.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:07 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:08.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:08 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/206353775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:08.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:08 smithi102 ceph-mon[36106]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-09T00:02:08.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:08 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/206353775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:08.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:08 smithi179 ceph-mon[36637]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-09T00:02:08.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:08 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/206353775' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:08.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:08 smithi073 ceph-mon[32309]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 130 KiB/s, 0 objects/s recovering 2024-03-09T00:02:11.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:10 smithi073 ceph-mon[32309]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:02:11.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:10 smithi102 ceph-mon[36106]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:02:11.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:10 smithi179 ceph-mon[36637]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 270 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:02:13.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:12 smithi102 ceph-mon[36106]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-09T00:02:13.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:12 smithi179 ceph-mon[36637]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-09T00:02:13.350 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:12 smithi073 ceph-mon[32309]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 78 KiB/s, 0 objects/s recovering 2024-03-09T00:02:15.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:14 smithi102 ceph-mon[36106]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-09T00:02:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:14 smithi179 ceph-mon[36637]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-09T00:02:15.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:14 smithi073 ceph-mon[32309]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 69 KiB/s, 0 objects/s recovering 2024-03-09T00:02:16.002 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-09T00:02:16.002 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:16.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-09T00:02:16.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-09T00:02:16.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:17.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:16 smithi073 ceph-mon[32309]: Deploying daemon osd.9 on smithi073 2024-03-09T00:02:17.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:16 smithi073 ceph-mon[32309]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-09T00:02:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:16 smithi102 ceph-mon[36106]: Deploying daemon osd.9 on smithi073 2024-03-09T00:02:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:16 smithi102 ceph-mon[36106]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-09T00:02:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:16 smithi179 ceph-mon[36637]: Deploying daemon osd.9 on smithi073 2024-03-09T00:02:17.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:16 smithi179 ceph-mon[36637]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-09T00:02:19.149 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:18 smithi073 ceph-mon[32309]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:18 smithi102 ceph-mon[36106]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:19.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:18 smithi179 ceph-mon[36637]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:20 smithi102 ceph-mon[36106]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:21.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:20 smithi179 ceph-mon[36637]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:21.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:20 smithi073 ceph-mon[32309]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:22.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:22.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:22 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:22 smithi073 ceph-mon[32309]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:22.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:22.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:22 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:22 smithi102 ceph-mon[36106]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:22.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:22.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:22 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:22.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:22 smithi179 ceph-mon[36637]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:24.945 INFO:teuthology.orchestra.run.smithi073.stdout:Created osd(s) 9 on host 'smithi073' 2024-03-09T00:02:24.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.949+0000 7f8fa5ffb700 1 -- 172.21.15.73:0/820463521 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f8fa8107e90 con 0x7f8f980a2910 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 msgr2=0x7f8f980a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 0x7f8f980a4dd0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8f9400a990 tx=0x7f8f94005c80 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 msgr2=0x7f8fa8196f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f8f9c00c010 tx=0x7f8f9c0047b0 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 shutdown_connections 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8fa8100d10 0x7f8fa8196f00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f980a2910 0x7f8f980a4dd0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8fa81028b0 0x7f8fa819e4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 --2- 172.21.15.73:0/820463521 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8fa8101f10 0x7f8fa8197440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:24.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.952+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 >> 172.21.15.73:0/820463521 conn(0x7f8fa80fc2d0 msgr2=0x7f8fa8105210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:02:24.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.953+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 shutdown_connections 2024-03-09T00:02:24.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:02:24.953+0000 7f8faf22e700 1 -- 172.21.15.73:0/820463521 wait complete. 2024-03-09T00:02:25.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:24 smithi073 ceph-mon[32309]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:25.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:24 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:24 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:24 smithi102 ceph-mon[36106]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:24 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:24 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:24 smithi179 ceph-mon[36637]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:24 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:24 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:25.572 DEBUG:teuthology.orchestra.run.smithi073:osd.9> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.9.service 2024-03-09T00:02:25.574 INFO:tasks.cephadm:Deploying osd.10 on smithi102 with /dev/vg_nvme/lv_1... 2024-03-09T00:02:25.575 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-09T00:02:25.817 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:02:26.255 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:25 smithi102 ceph-mon[36106]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-09T00:02:26.255 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:25 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:26.255 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:25 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:26.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:25 smithi179 ceph-mon[36637]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-09T00:02:26.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:25 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:26.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:25 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:26.352 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:25 smithi073 ceph-mon[32309]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-09T00:02:26.352 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:25 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:26.352 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:25 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:27.227 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:26 smithi102 ceph-mon[36106]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:27.227 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:26 smithi102 ceph-mon[36106]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-09T00:02:27.227 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:26 smithi102 ceph-mon[36106]: osdmap e65: 10 total, 9 up, 10 in 2024-03-09T00:02:27.227 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:26 smithi102 ceph-mon[36106]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:02:27.227 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:26 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:26 smithi179 ceph-mon[36637]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:26 smithi179 ceph-mon[36637]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-09T00:02:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:26 smithi179 ceph-mon[36637]: osdmap e65: 10 total, 9 up, 10 in 2024-03-09T00:02:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:26 smithi179 ceph-mon[36637]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:02:27.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:26 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:27.375 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:26 smithi073 ceph-mon[32309]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:27.376 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:26 smithi073 ceph-mon[32309]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-09T00:02:27.376 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:26 smithi073 ceph-mon[32309]: osdmap e65: 10 total, 9 up, 10 in 2024-03-09T00:02:27.376 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:26 smithi073 ceph-mon[32309]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]: dispatch 2024-03-09T00:02:27.376 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:26 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr: stderr: 10+0 records in 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:10+0 records out 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0113448 s, 924 MB/s 2024-03-09T00:02:27.896 INFO:teuthology.orchestra.run.smithi102.stderr:--> Zapping successful for: 2024-03-09T00:02:27.921 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:02:27.730+0000 7f0bade1f700 -1 osd.9 0 waiting for initial osdmap 2024-03-09T00:02:27.921 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:02:27.745+0000 7f0ba5c0a700 -1 osd.9 66 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:02:28.225 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:02:28.225 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: osdmap e66: 10 total, 9 up, 10 in 2024-03-09T00:02:28.225 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.226 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.226 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' 2024-03-09T00:02:28.226 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:27 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: osdmap e66: 10 total, 9 up, 10 in 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' 2024-03-09T00:02:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:27 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi073", "root=default"]}]': finished 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: osdmap e66: 10 total, 9 up, 10 in 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: from='osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248]' entity='osd.9' 2024-03-09T00:02:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:27 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:28.578 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi102:vg_nvme/lv_1 2024-03-09T00:02:28.821 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] boot 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: osdmap e67: 10 total, 10 up, 10 in 2024-03-09T00:02:29.113 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:28 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:29.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:02:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:02:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] boot 2024-03-09T00:02:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: osdmap e67: 10 total, 10 up, 10 in 2024-03-09T00:02:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:28 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:29.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:02:29.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:02:29.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 269 MiB used, 804 GiB / 805 GiB avail 2024-03-09T00:02:29.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: osd.9 [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] boot 2024-03-09T00:02:29.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: osdmap e67: 10 total, 10 up, 10 in 2024-03-09T00:02:29.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:28 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:02:30.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- 172.21.15.102:0/3308337674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514102f70 msgr2=0x7fe5141033f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:30.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 --2- 172.21.15.102:0/3308337674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514102f70 0x7fe5141033f0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fe508009990 tx=0x7fe508009ca0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- 172.21.15.102:0/3308337674 shutdown_connections 2024-03-09T00:02:30.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 --2- 172.21.15.102:0/3308337674 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 0x7fe514102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.627 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 --2- 172.21.15.102:0/3308337674 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514103930 0x7fe51410a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 --2- 172.21.15.102:0/3308337674 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514102f70 0x7fe5141033f0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- 172.21.15.102:0/3308337674 >> 172.21.15.102:0/3308337674 conn(0x7fe5140fd3f0 msgr2=0x7fe5140ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- 172.21.15.102:0/3308337674 shutdown_connections 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- 172.21.15.102:0/3308337674 wait complete. 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 Processor -- start 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.630+0000 7fe519dbd700 1 -- start start 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 0x7fe514198180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 0x7fe5141986c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51410dae0 con 0x7fe514101e10 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51410d960 con 0x7fe514103930 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe51410dc60 con 0x7fe514102f70 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:30.628 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.102:45858/0 (socket says 172.21.15.102:45858) 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 -- 172.21.15.102:0/2298485070 learned_addr learned my addr 172.21.15.102:0/2298485070 (peer_addr_for_me v2:172.21.15.102:0/0) 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe5137fe700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 0x7fe514198180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe512ffd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 0x7fe5141986c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 -- 172.21.15.102:0/2298485070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 msgr2=0x7fe5141986c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 0x7fe5141986c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 -- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 msgr2=0x7fe514198180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 0x7fe514198180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 -- 172.21.15.102:0/2298485070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe51419fde0 con 0x7fe514103930 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe512ffd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 0x7fe5141986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe513fff700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00b970 tx=0x7fe50c00bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c019070 con 0x7fe514103930 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5141a00d0 con 0x7fe514103930 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe50c00ce10 con 0x7fe514103930 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe50c0073d0 con 0x7fe514103930 2024-03-09T00:02:30.629 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.631+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5141a0640 con 0x7fe514103930 2024-03-09T00:02:30.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.633+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe514068aa0 con 0x7fe514103930 2024-03-09T00:02:30.630 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.633+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe50c01c070 con 0x7fe514103930 2024-03-09T00:02:30.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.634+0000 7fe510ff9700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 0x7fe4fc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:30.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.634+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fe50c0cd5e0 con 0x7fe514103930 2024-03-09T00:02:30.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.634+0000 7fe5137fe700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 0x7fe4fc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:30.631 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.635+0000 7fe5137fe700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 0x7fe4fc0a4d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe504009a10 tx=0x7fe504009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:30.633 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.639+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe50c09ae80 con 0x7fe514103930 2024-03-09T00:02:30.839 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:30.844+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe514062220 con 0x7fe4fc0a2890 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi073 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 4272M 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:30.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:30.903 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:30 smithi073 ceph-mon[32309]: osdmap e68: 10 total, 10 up, 10 in 2024-03-09T00:02:31.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi073 2024-03-09T00:02:31.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 4272M 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:31.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:30 smithi102 ceph-mon[36106]: osdmap e68: 10 total, 10 up, 10 in 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi073 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:31.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 4272M 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 304 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:31.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:30 smithi179 ceph-mon[36637]: osdmap e68: 10 total, 10 up, 10 in 2024-03-09T00:02:32.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:31 smithi179 ceph-mon[36637]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:32.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:31 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:32.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:31 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:32.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:31 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:32.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:31 smithi073 ceph-mon[32309]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:32.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:31 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:32.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:31 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:32.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:31 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:32.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:31 smithi102 ceph-mon[36106]: from='client.24314 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi102:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:32.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:31 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:32.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:31 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:32.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:31 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:33.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:32 smithi102 ceph-mon[36106]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:33.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:32 smithi179 ceph-mon[36637]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:33.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:32 smithi073 ceph-mon[32309]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:34.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:33 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/311110397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:33 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:33 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]': finished 2024-03-09T00:02:34.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:33 smithi179 ceph-mon[36637]: osdmap e69: 11 total, 10 up, 11 in 2024-03-09T00:02:34.083 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:33 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:34.111 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:33 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/311110397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.111 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:33 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.111 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:33 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]': finished 2024-03-09T00:02:34.111 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:33 smithi102 ceph-mon[36106]: osdmap e69: 11 total, 10 up, 11 in 2024-03-09T00:02:34.111 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:33 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:34.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:33 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/311110397' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:33 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]: dispatch 2024-03-09T00:02:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:33 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "9efc1e41-835a-496e-8487-b158cca2925a"}]': finished 2024-03-09T00:02:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:33 smithi073 ceph-mon[32309]: osdmap e69: 11 total, 10 up, 11 in 2024-03-09T00:02:34.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:33 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:35.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:34 smithi179 ceph-mon[36637]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:35.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:34 smithi179 ceph-mon[36637]: from='client.? 172.21.15.102:0/3394595831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:35.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:34 smithi073 ceph-mon[32309]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:35.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:34 smithi073 ceph-mon[32309]: from='client.? 172.21.15.102:0/3394595831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:35.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:34 smithi102 ceph-mon[36106]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:35.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:34 smithi102 ceph-mon[36106]: from='client.? 172.21.15.102:0/3394595831' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:02:37.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:36 smithi102 ceph-mon[36106]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:37.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:36 smithi179 ceph-mon[36637]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:37.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:36 smithi073 ceph-mon[32309]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:38 smithi102 ceph-mon[36106]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:38 smithi179 ceph-mon[36637]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:39.385 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:38 smithi073 ceph-mon[32309]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:41.151 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:40 smithi102 ceph-mon[36106]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:41.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:40 smithi179 ceph-mon[36637]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:41.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:40 smithi073 ceph-mon[32309]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:42.121 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:41 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-09T00:02:42.121 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:41 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:42.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:41 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-09T00:02:42.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:41 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:42.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:41 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-09T00:02:42.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:41 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:43.144 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:42 smithi102 ceph-mon[36106]: Deploying daemon osd.10 on smithi102 2024-03-09T00:02:43.145 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:42 smithi102 ceph-mon[36106]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:42 smithi179 ceph-mon[36637]: Deploying daemon osd.10 on smithi102 2024-03-09T00:02:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:42 smithi179 ceph-mon[36637]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:42 smithi073 ceph-mon[32309]: Deploying daemon osd.10 on smithi102 2024-03-09T00:02:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:42 smithi073 ceph-mon[32309]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:45.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:44 smithi102 ceph-mon[36106]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:44 smithi179 ceph-mon[36637]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:44 smithi073 ceph-mon[32309]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:46 smithi102 ceph-mon[36106]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:47.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:47.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:47.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:47.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:46 smithi179 ceph-mon[36637]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:47.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:47.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:47.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:47.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:46 smithi073 ceph-mon[32309]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:47.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:47.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:47.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:49.247 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:48 smithi102 ceph-mon[36106]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:49.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:48 smithi179 ceph-mon[36637]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:49.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:48 smithi073 ceph-mon[32309]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:51.006 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.010+0000 7fe510ff9700 1 -- 172.21.15.102:0/2298485070 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe514062220 con 0x7fe4fc0a2890 2024-03-09T00:02:51.006 INFO:teuthology.orchestra.run.smithi102.stdout:Created osd(s) 10 on host 'smithi102' 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 msgr2=0x7fe4fc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 0x7fe4fc0a4d50 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fe504009a10 tx=0x7fe504009190 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 msgr2=0x7fe51419f740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fe50c00b970 tx=0x7fe50c00bc80 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 shutdown_connections 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe514101e10 0x7fe514198180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe4fc0a2890 0x7fe4fc0a4d50 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe514103930 0x7fe51419f740 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 --2- 172.21.15.102:0/2298485070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe514102f70 0x7fe5141986c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:51.008 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 >> 172.21.15.102:0/2298485070 conn(0x7fe5140fd3f0 msgr2=0x7fe514106350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:02:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 shutdown_connections 2024-03-09T00:02:51.009 INFO:teuthology.orchestra.run.smithi102.stderr:2024-03-09T00:02:51.013+0000 7fe519dbd700 1 -- 172.21.15.102:0/2298485070 wait complete. 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:50 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:50 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:50 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:51.684 DEBUG:teuthology.orchestra.run.smithi102:osd.10> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.10.service 2024-03-09T00:02:51.686 INFO:tasks.cephadm:Deploying osd.11 on smithi179 with /dev/vg_nvme/lv_1... 2024-03-09T00:02:51.686 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-09T00:02:51.934 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: osdmap e70: 11 total, 10 up, 11 in 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:52.056 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:51 smithi179 ceph-mon[36637]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: osdmap e70: 11 total, 10 up, 11 in 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:52.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:51 smithi102 ceph-mon[36106]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:02:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-09T00:02:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: from='osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: osdmap e70: 11 total, 10 up, 11 in 2024-03-09T00:02:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:51 smithi073 ceph-mon[32309]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]: dispatch 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: osdmap e71: 11 total, 10 up, 11 in 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:52 smithi179 ceph-mon[36637]: from='osd.10 ' entity='osd.10' 2024-03-09T00:02:53.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:53.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:02:53.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: osdmap e71: 11 total, 10 up, 11 in 2024-03-09T00:02:53.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-mon[36106]: from='osd.10 ' entity='osd.10' 2024-03-09T00:02:53.260 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:02:52.874+0000 7f78935c1700 -1 osd.10 0 waiting for initial osdmap 2024-03-09T00:02:53.260 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:02:52 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:02:52.886+0000 7f788b3ac700 -1 osd.10 71 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi102", "root=default"]}]': finished 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: osdmap e71: 11 total, 10 up, 11 in 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:53.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:52 smithi073 ceph-mon[32309]: from='osd.10 ' entity='osd.10' 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr: stderr: 10+0 records in 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:10+0 records out 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0130945 s, 801 MB/s 2024-03-09T00:02:54.088 INFO:teuthology.orchestra.run.smithi179.stderr:--> Zapping successful for: 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] boot 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: osdmap e72: 11 total, 11 up, 11 in 2024-03-09T00:02:54.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:53 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] boot 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: osdmap e72: 11 total, 11 up, 11 in 2024-03-09T00:02:54.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:53 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:02:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:02:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: osd.10 [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] boot 2024-03-09T00:02:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: osdmap e72: 11 total, 11 up, 11 in 2024-03-09T00:02:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:53 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:02:54.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch daemon add osd smithi179:vg_nvme/lv_1 2024-03-09T00:02:55.020 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:02:55.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:54 smithi179 ceph-mon[36637]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:55.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:54 smithi179 ceph-mon[36637]: osdmap e73: 11 total, 11 up, 11 in 2024-03-09T00:02:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:54 smithi102 ceph-mon[36106]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:54 smithi102 ceph-mon[36106]: osdmap e73: 11 total, 11 up, 11 in 2024-03-09T00:02:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:54 smithi073 ceph-mon[32309]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 305 MiB used, 894 GiB / 894 GiB avail 2024-03-09T00:02:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:54 smithi073 ceph-mon[32309]: osdmap e73: 11 total, 11 up, 11 in 2024-03-09T00:02:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi102 2024-03-09T00:02:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.310 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 4270M 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.311 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:56 smithi179 ceph-mon[36637]: osdmap e74: 11 total, 11 up, 11 in 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi102 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 4270M 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:56 smithi073 ceph-mon[32309]: osdmap e74: 11 total, 11 up, 11 in 2024-03-09T00:02:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi102 2024-03-09T00:02:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 4270M 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:02:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:02:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:56 smithi102 ceph-mon[36106]: osdmap e74: 11 total, 11 up, 11 in 2024-03-09T00:02:56.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 -- 172.21.15.179:0/3852266498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100120 msgr2=0x7f6e98100540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:56.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 --2- 172.21.15.179:0/3852266498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100120 0x7f6e98100540 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f6e88009990 tx=0x7f6e88009ca0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 -- 172.21.15.179:0/3852266498 shutdown_connections 2024-03-09T00:02:56.888 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 --2- 172.21.15.179:0/3852266498 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 0x7f6e981020d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 --2- 172.21.15.179:0/3852266498 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100120 0x7f6e98100540 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 --2- 172.21.15.179:0/3852266498 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100e90 0x7f6e98101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 -- 172.21.15.179:0/3852266498 >> 172.21.15.179:0/3852266498 conn(0x7f6e980fba80 msgr2=0x7f6e980fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:02:56.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 -- 172.21.15.179:0/3852266498 shutdown_connections 2024-03-09T00:02:56.889 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.885+0000 7f6e9d058700 1 -- 172.21.15.179:0/3852266498 wait complete. 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 Processor -- start 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 -- start start 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100120 0x7f6e9819a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 0x7f6e981a17d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:56.890 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9810fca0 con 0x7f6e98101850 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9810fb20 con 0x7f6e98100120 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.886+0000 7f6e9d058700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e9810fe20 con 0x7f6e98100e90 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:51102/0 (socket says 172.21.15.179:51102) 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 -- 172.21.15.179:0/1342220613 learned_addr learned my addr 172.21.15.179:0/1342220613 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9759e700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 0x7f6e981a17d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:56.891 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e96d9d700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100120 0x7f6e9819a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 -- 172.21.15.179:0/1342220613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100120 msgr2=0x7f6e9819a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100120 0x7f6e9819a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 -- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 msgr2=0x7f6e981a17d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 0x7f6e981a17d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 -- 172.21.15.179:0/1342220613 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e981a1e70 con 0x7f6e98100e90 2024-03-09T00:02:56.892 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9659c700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00d9b0 tx=0x7f6e8c00dcc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c017070 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e981a2160 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e8c005020 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e8c009bd0 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.887+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e981a26a0 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.888+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e98068aa0 con 0x7f6e98100e90 2024-03-09T00:02:56.893 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.889+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6e8c0051d0 con 0x7f6e98100e90 2024-03-09T00:02:56.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.890+0000 7f6e7ffff700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 0x7f6e800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:02:56.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.890+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6694+0+0 (secure 0 0 0) 0x7f6e8c0ce2f0 con 0x7f6e98100e90 2024-03-09T00:02:56.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.890+0000 7f6e96d9d700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 0x7f6e800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:02:56.894 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.891+0000 7f6e96d9d700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 0x7f6e800a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800b5c0 tx=0x7f6e88005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:02:56.895 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:56.894+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6e8c020020 con 0x7f6e98100e90 2024-03-09T00:02:57.098 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:02:57.101+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e98062220 con 0x7f6e800a2890 2024-03-09T00:02:57.303 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:57 smithi073 ceph-mon[32309]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:02:57.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:57 smithi073 ceph-mon[32309]: osdmap e75: 11 total, 11 up, 11 in 2024-03-09T00:02:57.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:57 smithi179 ceph-mon[36637]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:02:57.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:57 smithi179 ceph-mon[36637]: osdmap e75: 11 total, 11 up, 11 in 2024-03-09T00:02:57.367 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:57 smithi102 ceph-mon[36106]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 336 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:02:57.367 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:57 smithi102 ceph-mon[36106]: osdmap e75: 11 total, 11 up, 11 in 2024-03-09T00:02:58.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:58.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:58.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:58.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:58 smithi102 ceph-mon[36106]: osdmap e76: 11 total, 11 up, 11 in 2024-03-09T00:02:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:58.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:58 smithi179 ceph-mon[36637]: osdmap e76: 11 total, 11 up, 11 in 2024-03-09T00:02:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-09T00:02:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-09T00:02:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:02:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:58 smithi073 ceph-mon[32309]: osdmap e76: 11 total, 11 up, 11 in 2024-03-09T00:02:59.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:59 smithi179 ceph-mon[36637]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:59.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:02:59 smithi179 ceph-mon[36637]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:02:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:59 smithi073 ceph-mon[32309]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:02:59 smithi073 ceph-mon[32309]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:02:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:59 smithi102 ceph-mon[36106]: from='client.24343 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi179:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:02:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:02:59 smithi102 ceph-mon[36106]: pgmap v243: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-09T00:03:00.316 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:00 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/3827173697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:00 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:00 smithi179 ceph-mon[36637]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]': finished 2024-03-09T00:03:00.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:00 smithi179 ceph-mon[36637]: osdmap e77: 12 total, 11 up, 12 in 2024-03-09T00:03:00.317 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:00 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/3827173697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:00 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:00 smithi073 ceph-mon[32309]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]': finished 2024-03-09T00:03:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:00 smithi073 ceph-mon[32309]: osdmap e77: 12 total, 11 up, 12 in 2024-03-09T00:03:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:00 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/3827173697' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:00 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]: dispatch 2024-03-09T00:03:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:00 smithi102 ceph-mon[36106]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "48124e49-5bc5-49ff-b7b4-c139fcb1346f"}]': finished 2024-03-09T00:03:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:00 smithi102 ceph-mon[36106]: osdmap e77: 12 total, 11 up, 12 in 2024-03-09T00:03:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:01.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:01 smithi179 ceph-mon[36637]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-09T00:03:01.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:01 smithi179 ceph-mon[36637]: from='client.? 172.21.15.179:0/2516626715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:03:01.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:01 smithi073 ceph-mon[32309]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-09T00:03:01.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:01 smithi073 ceph-mon[32309]: from='client.? 172.21.15.179:0/2516626715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:03:01.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:01 smithi102 ceph-mon[36106]: pgmap v244: 1 pgs: 1 active+undersized+remapped; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail; 102 KiB/s, 0 objects/s recovering 2024-03-09T00:03:01.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:01 smithi102 ceph-mon[36106]: from='client.? 172.21.15.179:0/2516626715' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-09T00:03:03.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:03 smithi179 ceph-mon[36637]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:03:03.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:03 smithi073 ceph-mon[32309]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:03:03.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:03 smithi102 ceph-mon[36106]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 338 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-09T00:03:05.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:05 smithi179 ceph-mon[36637]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:05.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:05 smithi073 ceph-mon[32309]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:05 smithi102 ceph-mon[36106]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:07.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:07 smithi179 ceph-mon[36637]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:07 smithi073 ceph-mon[32309]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:07 smithi102 ceph-mon[36106]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:08.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-09T00:03:08.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:08 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:08.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-09T00:03:08.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:08 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:08.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-09T00:03:08.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:08 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:09.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:09 smithi179 ceph-mon[36637]: Deploying daemon osd.11 on smithi179 2024-03-09T00:03:09.164 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:09 smithi179 ceph-mon[36637]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:09 smithi073 ceph-mon[32309]: Deploying daemon osd.11 on smithi179 2024-03-09T00:03:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:09 smithi073 ceph-mon[32309]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:09.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:09 smithi102 ceph-mon[36106]: Deploying daemon osd.11 on smithi179 2024-03-09T00:03:09.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:09 smithi102 ceph-mon[36106]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:11.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:11 smithi179 ceph-mon[36637]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:11.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:11 smithi073 ceph-mon[32309]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:11 smithi102 ceph-mon[36106]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:13.272 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:13 smithi179 ceph-mon[36637]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:13.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:13 smithi073 ceph-mon[32309]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:13.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:13 smithi102 ceph-mon[36106]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:14.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:14.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:14.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:14.799 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:14 smithi179 ceph-mon[36637]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:14.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:14.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:14.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:14.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:14 smithi073 ceph-mon[32309]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:15.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:14 smithi102 ceph-mon[36106]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:17.089 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:16 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:17.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:17.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:16 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:17.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:16 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:17.467 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.472+0000 7f6e7ffff700 1 -- 172.21.15.179:0/1342220613 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7f6e98062220 con 0x7f6e800a2890 2024-03-09T00:03:17.468 INFO:teuthology.orchestra.run.smithi179.stdout:Created osd(s) 11 on host 'smithi179' 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 msgr2=0x7f6e800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 0x7f6e800a4d50 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f6e8800b5c0 tx=0x7f6e88005fb0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 msgr2=0x7f6e9819a750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f6e8c00d9b0 tx=0x7f6e8c00dcc0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 shutdown_connections 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6e98101850 0x7f6e981a17d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.471 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6e800a2890 0x7f6e800a4d50 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6e98100120 0x7f6e9819a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 --2- 172.21.15.179:0/1342220613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6e98100e90 0x7f6e9819a750 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:17.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 >> 172.21.15.179:0/1342220613 conn(0x7f6e980fba80 msgr2=0x7f6e980fd580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:17.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 shutdown_connections 2024-03-09T00:03:17.472 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:17.476+0000 7f6e9d058700 1 -- 172.21.15.179:0/1342220613 wait complete. 2024-03-09T00:03:18.144 DEBUG:teuthology.orchestra.run.smithi179:osd.11> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.11.service 2024-03-09T00:03:18.156 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-09T00:03:18.157 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-09T00:03:18.409 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:03:18.532 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.533 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.533 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.533 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.533 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:18.533 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:18 smithi073 ceph-mon[32309]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:18.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:18 smithi102 ceph-mon[36106]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:18.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:18.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-09T00:03:18.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:18.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:18 smithi179 ceph-mon[36637]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:19.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:19 smithi179 ceph-mon[36637]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-09T00:03:19.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:19 smithi179 ceph-mon[36637]: osdmap e78: 12 total, 11 up, 12 in 2024-03-09T00:03:19.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:19 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:19.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:19 smithi179 ceph-mon[36637]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:19.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:19 smithi179 ceph-mon[36637]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:19.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:19 smithi102 ceph-mon[36106]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-09T00:03:19.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:19 smithi102 ceph-mon[36106]: osdmap e78: 12 total, 11 up, 12 in 2024-03-09T00:03:19.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:19 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:19.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:19 smithi102 ceph-mon[36106]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:19.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:19 smithi102 ceph-mon[36106]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:19.866 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:19 smithi073 ceph-mon[32309]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-09T00:03:19.866 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:19 smithi073 ceph-mon[32309]: osdmap e78: 12 total, 11 up, 12 in 2024-03-09T00:03:19.866 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:19 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:19.866 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:19 smithi073 ceph-mon[32309]: from='osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:19.867 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:19 smithi073 ceph-mon[32309]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]: dispatch 2024-03-09T00:03:20.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 -- 172.21.15.73:0/2014396710 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d0100d10 msgr2=0x7fb1d0101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:20.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/2014396710 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d0100d10 0x7fb1d0101130 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb1c00099c0 tx=0x7fb1c0009cd0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 -- 172.21.15.73:0/2014396710 shutdown_connections 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/2014396710 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d0100d10 0x7fb1d0101130 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/2014396710 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d01028b0 0x7fb1d0109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/2014396710 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 0x7fb1d0102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 -- 172.21.15.73:0/2014396710 >> 172.21.15.73:0/2014396710 conn(0x7fb1d00fc2b0 msgr2=0x7fb1d00fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 -- 172.21.15.73:0/2014396710 shutdown_connections 2024-03-09T00:03:20.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 -- 172.21.15.73:0/2014396710 wait complete. 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.288+0000 7fb1d4a40700 1 Processor -- start 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 -- start start 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d0100d10 0x7fb1d006a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 0x7fb1d019f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d010c8d0 con 0x7fb1d01028b0 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d010c750 con 0x7fb1d0100d10 2024-03-09T00:03:20.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1d4a40700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1d010ca50 con 0x7fb1d0101f10 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51612/0 (socket says 172.21.15.73:51612) 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 -- 172.21.15.73:0/1763833429 learned_addr learned my addr 172.21.15.73:0/1763833429 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ce59c700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d0100d10 0x7fb1d006a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1cdd9b700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 0x7fb1d019f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 -- 172.21.15.73:0/1763833429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 msgr2=0x7fb1d019f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 0x7fb1d019f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 -- 172.21.15.73:0/1763833429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d0100d10 msgr2=0x7fb1d006a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d0100d10 0x7fb1d006a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 -- 172.21.15.73:0/1763833429 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1d01a1f00 con 0x7fb1d01028b0 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1ced9d700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8007ee0 tx=0x7fb1c800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:20.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.289+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8004bb0 con 0x7fb1d01028b0 2024-03-09T00:03:20.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.290+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1d01a21f0 con 0x7fb1d01028b0 2024-03-09T00:03:20.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.290+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb1c8004d50 con 0x7fb1d01028b0 2024-03-09T00:03:20.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.290+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb1c8005960 con 0x7fb1d01028b0 2024-03-09T00:03:20.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.290+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1d01a26e0 con 0x7fb1d01028b0 2024-03-09T00:03:20.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.290+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1d0068aa0 con 0x7fb1d01028b0 2024-03-09T00:03:20.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.292+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb1c801a020 con 0x7fb1d01028b0 2024-03-09T00:03:20.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.293+0000 7fb1bf7fe700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 0x7fb1b80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:20.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.293+0000 7fb1ce59c700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 0x7fb1b80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:20.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.293+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 6847+0+0 (secure 0 0 0) 0x7fb1c80ce1c0 con 0x7fb1d01028b0 2024-03-09T00:03:20.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.293+0000 7fb1ce59c700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 0x7fb1b80a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0000c00 tx=0x7fb1c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:20.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.297+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb1c8010070 con 0x7fb1d01028b0 2024-03-09T00:03:20.456 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:03:20.183+0000 7fb79a89d700 -1 osd.11 0 waiting for initial osdmap 2024-03-09T00:03:20.456 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:03:20.196+0000 7fb792688700 -1 osd.11 79 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-09T00:03:20.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.504+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fb1d002d8d0 con 0x7fb1d01028b0 2024-03-09T00:03:20.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.504+0000 7fb1bf7fe700 1 -- 172.21.15.73:0/1763833429 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v79) v1 ==== 74+0+133 (secure 0 0 0) 0x7fb1d002d8d0 con 0x7fb1d01028b0 2024-03-09T00:03:20.503 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:20.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 msgr2=0x7fb1b80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 0x7fb1b80a4d50 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb1c0000c00 tx=0x7fb1c0005fb0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 msgr2=0x7fb1d01a1720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fb1c8007ee0 tx=0x7fb1c800d3b0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 shutdown_connections 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb1d01028b0 0x7fb1d01a1720 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb1b80a2890 0x7fb1b80a4d50 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb1d0100d10 0x7fb1d006a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 --2- 172.21.15.73:0/1763833429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb1d0101f10 0x7fb1d019f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:20.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 >> 172.21.15.73:0/1763833429 conn(0x7fb1d00fc2b0 msgr2=0x7fb1d0105330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:20.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 shutdown_connections 2024-03-09T00:03:20.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:20.506+0000 7fb1d4a40700 1 -- 172.21.15.73:0/1763833429 wait complete. 2024-03-09T00:03:20.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:03:20.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: osdmap e79: 12 total, 11 up, 12 in 2024-03-09T00:03:20.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:20.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:20 smithi102 ceph-mon[36106]: from='osd.11 ' entity='osd.11' 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: osdmap e79: 12 total, 11 up, 12 in 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:20.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:20 smithi179 ceph-mon[36637]: from='osd.11 ' entity='osd.11' 2024-03-09T00:03:20.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi179", "root=default"]}]': finished 2024-03-09T00:03:20.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: osdmap e79: 12 total, 11 up, 12 in 2024-03-09T00:03:20.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.878 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:20.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 337 MiB used, 983 GiB / 983 GiB avail 2024-03-09T00:03:20.879 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:20 smithi073 ceph-mon[32309]: from='osd.11 ' entity='osd.11' 2024-03-09T00:03:21.340 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":79,"num_osds":12,"num_up_osds":11,"osd_up_since":1709942573,"num_in_osds":12,"osd_in_since":1709942579,"num_remapped_pgs":0} 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: purged_snaps scrub starts 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: purged_snaps scrub ok 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1763833429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] boot 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: osdmap e80: 12 total, 12 up, 12 in 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:21.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:21 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: purged_snaps scrub starts 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: purged_snaps scrub ok 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1763833429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] boot 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: osdmap e80: 12 total, 12 up, 12 in 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:21.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:21 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: purged_snaps scrub starts 2024-03-09T00:03:21.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: purged_snaps scrub ok 2024-03-09T00:03:21.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1763833429' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: osd.11 [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] boot 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: osdmap e80: 12 total, 12 up, 12 in 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:21.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:21.761 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:21 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:22.342 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd stat -f json 2024-03-09T00:03:22.598 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:03:22.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:22 smithi073 ceph-mon[32309]: Detected new or changed devices on smithi179 2024-03-09T00:03:22.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:22 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 4270M 2024-03-09T00:03:22.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:22 smithi073 ceph-mon[32309]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:22.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:22 smithi102 ceph-mon[36106]: Detected new or changed devices on smithi179 2024-03-09T00:03:22.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:22 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 4270M 2024-03-09T00:03:22.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:22 smithi102 ceph-mon[36106]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:22.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:22 smithi179 ceph-mon[36637]: Detected new or changed devices on smithi179 2024-03-09T00:03:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:22 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 4270M 2024-03-09T00:03:22.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:22 smithi179 ceph-mon[36637]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:23.735 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:23 smithi073 ceph-mon[32309]: osdmap e81: 12 total, 12 up, 12 in 2024-03-09T00:03:23.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:23 smithi102 ceph-mon[36106]: osdmap e81: 12 total, 12 up, 12 in 2024-03-09T00:03:23.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:23 smithi179 ceph-mon[36637]: osdmap e81: 12 total, 12 up, 12 in 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 -- 172.21.15.73:0/3380141230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 msgr2=0x7fe710103550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 --2- 172.21.15.73:0/3380141230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe710103550 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe704009990 tx=0x7fe704009ca0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 -- 172.21.15.73:0/3380141230 shutdown_connections 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 --2- 172.21.15.73:0/3380141230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe71010a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 --2- 172.21.15.73:0/3380141230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 0x7fe710100530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 --2- 172.21.15.73:0/3380141230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe710103550 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 -- 172.21.15.73:0/3380141230 >> 172.21.15.73:0/3380141230 conn(0x7fe7100fba70 msgr2=0x7fe7100fdeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:24.344 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 -- 172.21.15.73:0/3380141230 shutdown_connections 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.345+0000 7fe716704700 1 -- 172.21.15.73:0/3380141230 wait complete. 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.346+0000 7fe716704700 1 Processor -- start 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.346+0000 7fe716704700 1 -- start start 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.346+0000 7fe716704700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 0x7fe710102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.346+0000 7fe716704700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.346+0000 7fe716704700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe7101015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe716704700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71010da80 con 0x7fe7101030f0 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe716704700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71010d900 con 0x7fe710100110 2024-03-09T00:03:24.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe716704700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe71010dc00 con 0x7fe710103a90 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51634/0 (socket says 172.21.15.73:51634) 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 -- 172.21.15.73:0/1915662677 learned_addr learned my addr 172.21.15.73:0/1915662677 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70ffff700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 0x7fe710102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe714ca1700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe7101015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 -- 172.21.15.73:0/1915662677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 msgr2=0x7fe7101015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe7101015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 -- 172.21.15.73:0/1915662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 msgr2=0x7fe710102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 0x7fe710102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 -- 172.21.15.73:0/1915662677 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe710101e90 con 0x7fe7101030f0 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe714ca1700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe7101015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:03:24.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70f7fe700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe704005910 tx=0x7fe7040047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe704027070 con 0x7fe7101030f0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe704022470 con 0x7fe7101030f0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7101a2190 con 0x7fe7101030f0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.347+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe70400f6c0 con 0x7fe7101030f0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.348+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7101a2660 con 0x7fe7101030f0 2024-03-09T00:03:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.349+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe710068aa0 con 0x7fe7101030f0 2024-03-09T00:03:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.350+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe70402a070 con 0x7fe7101030f0 2024-03-09T00:03:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.350+0000 7fe70d7fa700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 0x7fe6f80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.351+0000 7fe70ffff700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 0x7fe6f80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.351+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe7040cf540 con 0x7fe7101030f0 2024-03-09T00:03:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.351+0000 7fe70ffff700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 0x7fe6f80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe71006a320 tx=0x7fe70000a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:24.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.355+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe704013070 con 0x7fe7101030f0 2024-03-09T00:03:24.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.552+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fe7100629f0 con 0x7fe7101030f0 2024-03-09T00:03:24.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.552+0000 7fe70d7fa700 1 -- 172.21.15.73:0/1915662677 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7fe70409cb10 con 0x7fe7101030f0 2024-03-09T00:03:24.547 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:24.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 msgr2=0x7fe6f80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 0x7fe6f80a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fe71006a320 tx=0x7fe70000a380 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 msgr2=0x7fe7101010a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe704005910 tx=0x7fe7040047b0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 shutdown_connections 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe6f80a2890 0x7fe6f80a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe710103a90 0x7fe7101015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe710100110 0x7fe710102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 --2- 172.21.15.73:0/1915662677 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe7101030f0 0x7fe7101010a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 >> 172.21.15.73:0/1915662677 conn(0x7fe7100fba70 msgr2=0x7fe7101063f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 shutdown_connections 2024-03-09T00:03:24.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:24.554+0000 7fe716704700 1 -- 172.21.15.73:0/1915662677 wait complete. 2024-03-09T00:03:24.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:24 smithi073 ceph-mon[32309]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:24.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:24 smithi102 ceph-mon[36106]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:24.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:24 smithi179 ceph-mon[36637]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:25.194 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1709942601,"num_in_osds":12,"osd_in_since":1709942579,"num_remapped_pgs":0} 2024-03-09T00:03:25.194 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-09T00:03:25.431 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:03:25.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:25 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1915662677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:25.820 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:25 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1915662677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:25.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:25 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1915662677' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-09T00:03:26.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:26 smithi073 ceph-mon[32309]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:26.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:26 smithi102 ceph-mon[36106]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:26.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:26 smithi179 ceph-mon[36637]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:27.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 -- 172.21.15.73:0/1093479676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938100ce0 msgr2=0x7fd938101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:27.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/1093479676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938100ce0 0x7fd938101140 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fd928009990 tx=0x7fd928009ca0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 -- 172.21.15.73:0/1093479676 shutdown_connections 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/1093479676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938101680 0x7fd938069a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/1093479676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938106d20 0x7fd938107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/1093479676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938100ce0 0x7fd938101140 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 -- 172.21.15.73:0/1093479676 >> 172.21.15.73:0/1093479676 conn(0x7fd938077c10 msgr2=0x7fd938078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 -- 172.21.15.73:0/1093479676 shutdown_connections 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 -- 172.21.15.73:0/1093479676 wait complete. 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.312+0000 7fd93d2d7700 1 Processor -- start 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 -- start start 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938100ce0 0x7fd938109c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:27.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938101680 0x7fd93810a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9381120f0 con 0x7fd938106d20 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938111f70 con 0x7fd938100ce0 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd93d2d7700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd938112270 con 0x7fd938101680 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51664/0 (socket says 172.21.15.73:51664) 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 -- 172.21.15.73:0/3329828380 learned_addr learned my addr 172.21.15.73:0/3329828380 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd936ffd700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938100ce0 0x7fd938109c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9367fc700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938101680 0x7fd93810a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:27.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 -- 172.21.15.73:0/3329828380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938101680 msgr2=0x7fd93810a150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938101680 0x7fd93810a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 -- 172.21.15.73:0/3329828380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938100ce0 msgr2=0x7fd938109c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938100ce0 0x7fd938109c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 -- 172.21.15.73:0/3329828380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9381a6b40 con 0x7fd938106d20 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.313+0000 7fd9377fe700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd930007ee0 tx=0x7fd93000d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.314+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd93000dbd0 con 0x7fd938106d20 2024-03-09T00:03:27.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.314+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9381a6e30 con 0x7fd938106d20 2024-03-09T00:03:27.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.314+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd93000dd70 con 0x7fd938106d20 2024-03-09T00:03:27.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.314+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd930005a60 con 0x7fd938106d20 2024-03-09T00:03:27.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.314+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9381a7390 con 0x7fd938106d20 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.315+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd938068aa0 con 0x7fd938106d20 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.316+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd93001a020 con 0x7fd938106d20 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.316+0000 7fd92ffff700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 0x7fd9240a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.316+0000 7fd936ffd700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 0x7fd9240a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.316+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd9300cdc90 con 0x7fd938106d20 2024-03-09T00:03:27.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.317+0000 7fd936ffd700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 0x7fd9240a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd920009a10 tx=0x7fd920009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:27.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.321+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd930010070 con 0x7fd938106d20 2024-03-09T00:03:27.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.514+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd93802d830 con 0x7fd938106d20 2024-03-09T00:03:27.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.516+0000 7fd92ffff700 1 -- 172.21.15.73:0/3329828380 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18443 (secure 0 0 0) 0x7fd93009b1d0 con 0x7fd938106d20 2024-03-09T00:03:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:27.511 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":81,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:03:22.460611+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:03:27.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 msgr2=0x7fd9240a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:27.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 0x7fd9240a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fd920009a10 tx=0x7fd920009190 comp rx=0 tx=0).stop 2024-03-09T00:03:27.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 msgr2=0x7fd93810a690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:27.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fd930007ee0 tx=0x7fd93000d3b0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 shutdown_connections 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd938106d20 0x7fd93810a690 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd9240a2890 0x7fd9240a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd938100ce0 0x7fd938109c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 --2- 172.21.15.73:0/3329828380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd938101680 0x7fd93810a150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 >> 172.21.15.73:0/3329828380 conn(0x7fd938077c10 msgr2=0x7fd93810bd50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 shutdown_connections 2024-03-09T00:03:27.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:27.518+0000 7fd93d2d7700 1 -- 172.21.15.73:0/3329828380 wait complete. 2024-03-09T00:03:27.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:27 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3329828380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:27.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3329828380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:28.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:27 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3329828380' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:28.189 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-09T00:03:28.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-09T00:03:28.428 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:03:28.831 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:28 smithi073 ceph-mon[32309]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:28.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:28 smithi179 ceph-mon[36637]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:29.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:28 smithi102 ceph-mon[36106]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 -- 172.21.15.73:0/731498779 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 msgr2=0x7f4a60105920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/731498779 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60105920 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c009990 tx=0x7f4a5c009ca0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 -- 172.21.15.73:0/731498779 shutdown_connections 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/731498779 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60106000 0x7f4a6010dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/731498779 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60100f70 0x7f4a60101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/731498779 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60105920 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 -- 172.21.15.73:0/731498779 >> 172.21.15.73:0/731498779 conn(0x7f4a60076d00 msgr2=0x7f4a60077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 -- 172.21.15.73:0/731498779 shutdown_connections 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.159+0000 7f4a67e6e700 1 -- 172.21.15.73:0/731498779 wait complete. 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 Processor -- start 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 -- start start 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60100f70 0x7f4a60198600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60106000 0x7f4a6019ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60111020 con 0x7f4a60101920 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a60110ea0 con 0x7f4a60106000 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a67e6e700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a601111a0 con 0x7f4a60100f70 2024-03-09T00:03:30.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a65409700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a65409700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33904/0 (socket says 172.21.15.73:33904) 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a65409700 1 -- 172.21.15.73:0/991281631 learned_addr learned my addr 172.21.15.73:0/991281631 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a65c0a700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60100f70 0x7f4a60198600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.160+0000 7f4a6640b700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60106000 0x7f4a6019ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 -- 172.21.15.73:0/991281631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60100f70 msgr2=0x7f4a60198600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60100f70 0x7f4a60198600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 -- 172.21.15.73:0/991281631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60106000 msgr2=0x7f4a6019ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60106000 0x7f4a6019ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 -- 172.21.15.73:0/991281631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a6019d610 con 0x7f4a60101920 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a65409700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000c00 tx=0x7f4a5c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:30.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c027070 con 0x7f4a60101920 2024-03-09T00:03:30.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4a5c004050 con 0x7f4a60101920 2024-03-09T00:03:30.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a6019d8a0 con 0x7f4a60101920 2024-03-09T00:03:30.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4a5c00f460 con 0x7f4a60101920 2024-03-09T00:03:30.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.161+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a601a5180 con 0x7f4a60101920 2024-03-09T00:03:30.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.162+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a60102400 con 0x7f4a60101920 2024-03-09T00:03:30.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.163+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4a5c02a070 con 0x7f4a60101920 2024-03-09T00:03:30.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.164+0000 7f4a56ffd700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 0x7f4a4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:30.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.164+0000 7f4a65c0a700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 0x7f4a4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:30.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.164+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f4a5c0cf930 con 0x7f4a60101920 2024-03-09T00:03:30.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.164+0000 7f4a65c0a700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 0x7f4a4c0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4a50005950 tx=0x7f4a500058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:30.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.168+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4a5c031070 con 0x7f4a60101920 2024-03-09T00:03:30.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.359+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4a60199960 con 0x7f4a60101920 2024-03-09T00:03:30.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.359+0000 7f4a56ffd700 1 -- 172.21.15.73:0/991281631 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4a5c013070 con 0x7f4a60101920 2024-03-09T00:03:30.355 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 msgr2=0x7f4a4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 0x7f4a4c0a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f4a50005950 tx=0x7f4a500058e0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 msgr2=0x7f4a60198b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f4a5c000c00 tx=0x7f4a5c0047b0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 shutdown_connections 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4a4c0a2890 0x7f4a4c0a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4a60106000 0x7f4a6019ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4a60100f70 0x7f4a60198600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 --2- 172.21.15.73:0/991281631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4a60101920 0x7f4a60198b40 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:30.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 >> 172.21.15.73:0/991281631 conn(0x7f4a60076d00 msgr2=0x7f4a6006a860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:30.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 shutdown_connections 2024-03-09T00:03:30.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:30.361+0000 7f4a67e6e700 1 -- 172.21.15.73:0/991281631 wait complete. 2024-03-09T00:03:31.030 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:30 smithi073 ceph-mon[32309]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:31.031 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/991281631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:31.061 INFO:tasks.cephadm:Adding mds.a on smithi073 2024-03-09T00:03:31.061 INFO:tasks.cephadm:Adding mds.d on smithi073 2024-03-09T00:03:31.061 INFO:tasks.cephadm:Adding mds.g on smithi073 2024-03-09T00:03:31.061 INFO:tasks.cephadm:Adding mds.j on smithi073 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.b on smithi102 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.e on smithi102 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.h on smithi102 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.k on smithi102 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.c on smithi179 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.f on smithi179 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.i on smithi179 2024-03-09T00:03:31.062 INFO:tasks.cephadm:Adding mds.l on smithi179 2024-03-09T00:03:31.062 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph orch apply mds all '12;smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l' 2024-03-09T00:03:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:30 smithi102 ceph-mon[36106]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/991281631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:31.311 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:03:31.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:30 smithi179 ceph-mon[36637]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:31.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/991281631' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:33.015 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:32 smithi179 ceph-mon[36637]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:33.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- 172.21.15.179:0/1538111322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4101e10 msgr2=0x7ff9c4102230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:33.122 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 --2- 172.21.15.179:0/1538111322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4101e10 0x7ff9c4102230 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9c0009990 tx=0x7ff9c0009ca0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- 172.21.15.179:0/1538111322 shutdown_connections 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 --2- 172.21.15.179:0/1538111322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4103930 0x7ff9c410a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 --2- 172.21.15.179:0/1538111322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4101e10 0x7ff9c4102230 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 --2- 172.21.15.179:0/1538111322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4102f70 0x7ff9c41033f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- 172.21.15.179:0/1538111322 >> 172.21.15.179:0/1538111322 conn(0x7ff9c40fd3f0 msgr2=0x7ff9c40ff810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- 172.21.15.179:0/1538111322 shutdown_connections 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- 172.21.15.179:0/1538111322 wait complete. 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 Processor -- start 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 -- start start 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.125+0000 7ff9cb508700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4101e10 0x7ff9c4198010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 0x7ff9c4198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:33.123 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c410da20 con 0x7ff9c4102f70 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c410d8a0 con 0x7ff9c4101e10 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9c410dba0 con 0x7ff9c4103930 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.179:35142/0 (socket says 172.21.15.179:35142) 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 -- 172.21.15.179:0/3741778193 learned_addr learned my addr 172.21.15.179:0/3741778193 (peer_addr_for_me v2:172.21.15.179:0/0) 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c92a4700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4101e10 0x7ff9c4198010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c8aa3700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 0x7ff9c4198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 -- 172.21.15.179:0/3741778193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4101e10 msgr2=0x7ff9c4198010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4101e10 0x7ff9c4198010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 -- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 msgr2=0x7ff9c4198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 0x7ff9c4198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 -- 172.21.15.179:0/3741778193 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9c419fc70 con 0x7ff9c4103930 2024-03-09T00:03:33.124 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c8aa3700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 0x7ff9c4198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9c9aa5700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00b9a0 tx=0x7ff9bc00bd60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc019070 con 0x7ff9c4103930 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9bc014440 con 0x7ff9c4103930 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9c419ff60 con 0x7ff9c4103930 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9bc0075d0 con 0x7ff9c4103930 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.126+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9c41a0440 con 0x7ff9c4103930 2024-03-09T00:03:33.125 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.127+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9c4068aa0 con 0x7ff9c4103930 2024-03-09T00:03:33.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.128+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff9bc01c070 con 0x7ff9c4103930 2024-03-09T00:03:33.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.129+0000 7ff9ba7fc700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 0x7ff9b00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:33.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.129+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff9bc0cdea0 con 0x7ff9c4103930 2024-03-09T00:03:33.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.129+0000 7ff9c92a4700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 0x7ff9b00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:33.126 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.130+0000 7ff9c92a4700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 0x7ff9b00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:33.127 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.132+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff9bc09b470 con 0x7ff9c4103930 2024-03-09T00:03:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:32 smithi102 ceph-mon[36106]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:33.330 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.336+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}) v1 -- 0x7ff9c4062320 con 0x7ff9b00a2890 2024-03-09T00:03:33.339 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.344+0000 7ff9ba7fc700 1 -- 172.21.15.179:0/3741778193 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7ff9c4062320 con 0x7ff9b00a2890 2024-03-09T00:03:33.339 INFO:teuthology.orchestra.run.smithi179.stdout:Scheduled mds.all update... 2024-03-09T00:03:33.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.346+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 msgr2=0x7ff9b00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:33.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.346+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 0x7ff9b00a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7ff9c000b5c0 tx=0x7ff9c0005fb0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.341 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.346+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 msgr2=0x7ff9c419f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.346+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7ff9bc00b9a0 tx=0x7ff9bc00bd60 comp rx=0 tx=0).stop 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 shutdown_connections 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9b00a2890 0x7ff9b00a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff9c4103930 0x7ff9c419f5d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff9c4101e10 0x7ff9c4198010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 --2- 172.21.15.179:0/3741778193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff9c4102f70 0x7ff9c4198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 >> 172.21.15.179:0/3741778193 conn(0x7ff9c40fd3f0 msgr2=0x7ff9c4106300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 shutdown_connections 2024-03-09T00:03:33.342 INFO:teuthology.orchestra.run.smithi179.stderr:2024-03-09T00:03:33.347+0000 7ff9cb508700 1 -- 172.21.15.179:0/3741778193 wait complete. 2024-03-09T00:03:33.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:32 smithi073 ceph-mon[32309]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:34.041 DEBUG:teuthology.orchestra.run.smithi073:mds.a> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.a.service 2024-03-09T00:03:34.043 DEBUG:teuthology.orchestra.run.smithi073:mds.d> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.d.service 2024-03-09T00:03:34.087 DEBUG:teuthology.orchestra.run.smithi073:mds.g> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.g.service 2024-03-09T00:03:34.130 DEBUG:teuthology.orchestra.run.smithi073:mds.j> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.j.service 2024-03-09T00:03:34.173 DEBUG:teuthology.orchestra.run.smithi102:mds.b> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.b.service 2024-03-09T00:03:34.175 DEBUG:teuthology.orchestra.run.smithi102:mds.e> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.e.service 2024-03-09T00:03:34.218 DEBUG:teuthology.orchestra.run.smithi102:mds.h> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.h.service 2024-03-09T00:03:34.261 DEBUG:teuthology.orchestra.run.smithi102:mds.k> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.k.service 2024-03-09T00:03:34.304 DEBUG:teuthology.orchestra.run.smithi179:mds.c> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.c.service 2024-03-09T00:03:34.306 DEBUG:teuthology.orchestra.run.smithi179:mds.f> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.f.service 2024-03-09T00:03:34.349 DEBUG:teuthology.orchestra.run.smithi179:mds.i> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.i.service 2024-03-09T00:03:34.392 DEBUG:teuthology.orchestra.run.smithi179:mds.l> sudo journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.l.service 2024-03-09T00:03:34.435 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-09T00:03:34.436 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:03:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:03:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: Saving service mds.all spec with placement smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:34 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 -- 172.21.15.73:0/3553158463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 msgr2=0x7f47c0109460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/3553158463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c0109460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f47b8009ca0 tx=0x7f47b802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 -- 172.21.15.73:0/3553158463 shutdown_connections 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/3553158463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c0109460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/3553158463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/3553158463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 0x7f47c0104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 -- 172.21.15.73:0/3553158463 >> 172.21.15.73:0/3553158463 conn(0x7f47c00fb780 msgr2=0x7f47c00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 -- 172.21.15.73:0/3553158463 shutdown_connections 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 -- 172.21.15.73:0/3553158463 wait complete. 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.545+0000 7f47c5d1b640 1 Processor -- start 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- start start 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 0x7f47c01975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010cfa0 con 0x7f47c0107070 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010ce20 con 0x7f47c0104740 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c010d120 con 0x7f47c0101e10 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33924/0 (socket says 172.21.15.73:33924) 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 -- 172.21.15.73:0/2004601906 learned_addr learned my addr 172.21.15.73:0/2004601906 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bf7fe640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 0x7f47c01975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47beffd640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 -- 172.21.15.73:0/2004601906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 msgr2=0x7f47c01975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 0x7f47c01975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 -- 172.21.15.73:0/2004601906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 msgr2=0x7f47c0197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 -- 172.21.15.73:0/2004601906 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c019f210 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47beffd640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bffff640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8037040 tx=0x7f47b8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8041020 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c019f4a0 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f47b8034d90 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f47b8039040 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.546+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c019f990 con 0x7f47c0107070 2024-03-09T00:03:34.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.547+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c00fd270 con 0x7f47c0107070 2024-03-09T00:03:34.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.548+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f47b8044020 con 0x7f47c0107070 2024-03-09T00:03:34.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.549+0000 7f47bcff9640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 0x7f47ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.549+0000 7f47bf7fe640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 0x7f47ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.549+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f47b80f0540 con 0x7f47c0107070 2024-03-09T00:03:34.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.549+0000 7f47bf7fe640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 0x7f47ac0a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47a8005fd0 tx=0x7f47a800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:34.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.553+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f47b804b020 con 0x7f47c0107070 2024-03-09T00:03:34.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: Saving service mds.all spec with placement smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:34.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:34 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='client.24382 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:03:34.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: Saving service mds.all spec with placement smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-03-09T00:03:34.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:34.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:34 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:34.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.731+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f47c0061ef0 con 0x7f47c0107070 2024-03-09T00:03:34.727 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.732+0000 7f47bcff9640 1 -- 172.21.15.73:0/2004601906 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18443 (secure 0 0 0) 0x7f47b80bda80 con 0x7f47c0107070 2024-03-09T00:03:34.727 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:34.727 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":81,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:03:22.460611+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 msgr2=0x7f47ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 0x7f47ac0a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f47a8005fd0 tx=0x7f47a800b390 comp rx=0 tx=0).stop 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 msgr2=0x7f47c019eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f47b8037040 tx=0x7f47b8032f20 comp rx=0 tx=0).stop 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 shutdown_connections 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f47ac0a2800 0x7f47ac0a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f47c0107070 0x7f47c019eba0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f47c0104740 0x7f47c0197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 --2- 172.21.15.73:0/2004601906 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f47c0101e10 0x7f47c01975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 >> 172.21.15.73:0/2004601906 conn(0x7f47c00fb780 msgr2=0x7f47c0102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 shutdown_connections 2024-03-09T00:03:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.734+0000 7f47c5d1b640 1 -- 172.21.15.73:0/2004601906 wait complete. 2024-03-09T00:03:34.740 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-09T00:03:34.740 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.849+0000 7f1fa02ab640 1 -- 172.21.15.73:0/1076660426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102510 msgr2=0x7f1f98102990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.849+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/1076660426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102510 0x7f1f98102990 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f1f88009900 tx=0x7f1f8802bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- 172.21.15.73:0/1076660426 shutdown_connections 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/1076660426 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102ed0 0x7f1f98109760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/1076660426 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102510 0x7f1f98102990 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/1076660426 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 0x7f1f98101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- 172.21.15.73:0/1076660426 >> 172.21.15.73:0/1076660426 conn(0x7f1f980fc890 msgr2=0x7f1f980fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- 172.21.15.73:0/1076660426 shutdown_connections 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- 172.21.15.73:0/1076660426 wait complete. 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 Processor -- start 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- start start 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 0x7f1f98197670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 0x7f1f98197bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810cfb0 con 0x7f1f98102ed0 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810ce30 con 0x7f1f98102510 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1fa02ab640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f9810d130 con 0x7f1f98101320 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1f9e848640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1f9e848640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33932/0 (socket says 172.21.15.73:33932) 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.850+0000 7f1f9e848640 1 -- 172.21.15.73:0/3325893397 learned_addr learned my addr 172.21.15.73:0/3325893397 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e047640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 0x7f1f98197670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9d846640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 0x7f1f98197bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 -- 172.21.15.73:0/3325893397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 msgr2=0x7f1f98197670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 0x7f1f98197670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 -- 172.21.15.73:0/3325893397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 msgr2=0x7f1f98197bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 0x7f1f98197bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 -- 172.21.15.73:0/3325893397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f9819f290 con 0x7f1f98102ed0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9d846640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 0x7f1f98197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f9e848640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000efd0 tx=0x7f1f9000c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f90003b70 con 0x7f1f98102ed0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1f90003d10 con 0x7f1f98102ed0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f9819f580 con 0x7f1f98102ed0 2024-03-09T00:03:34.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1f900109f0 con 0x7f1f98102ed0 2024-03-09T00:03:34.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.851+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f9819fa80 con 0x7f1f98102ed0 2024-03-09T00:03:34.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.852+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f98108660 con 0x7f1f98102ed0 2024-03-09T00:03:34.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.853+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1f90010b90 con 0x7f1f98102ed0 2024-03-09T00:03:34.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.854+0000 7f1f8f7fe640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 0x7f1f840a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:34.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.854+0000 7f1f9e047640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 0x7f1f840a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:34.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.854+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1f900cee70 con 0x7f1f98102ed0 2024-03-09T00:03:34.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.854+0000 7f1f9e047640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 0x7f1f840a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f94005fd0 tx=0x7f1f9400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:34.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:34.857+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1f9009c3b0 con 0x7f1f98102ed0 2024-03-09T00:03:35.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.030+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f1f9802ce70 con 0x7f1f98102ed0 2024-03-09T00:03:35.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.031+0000 7f1f8f7fe640 1 -- 172.21.15.73:0/3325893397 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f1f9009bcb0 con 0x7f1f98102ed0 2024-03-09T00:03:35.026 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 msgr2=0x7f1f840a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 0x7f1f840a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f1f94005fd0 tx=0x7f1f9400b390 comp rx=0 tx=0).stop 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 msgr2=0x7f1f9819ec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f1f9000efd0 tx=0x7f1f9000c6a0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 shutdown_connections 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1f840a2800 0x7f1f840a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1f98102ed0 0x7f1f9819ec20 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1f98102510 0x7f1f98197bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 --2- 172.21.15.73:0/3325893397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1f98101320 0x7f1f98197670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 >> 172.21.15.73:0/3325893397 conn(0x7f1f980fc890 msgr2=0x7f1f98105830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:35.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 shutdown_connections 2024-03-09T00:03:35.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.033+0000 7f1fa02ab640 1 -- 172.21.15.73:0/3325893397 wait complete. 2024-03-09T00:03:35.039 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 -- 172.21.15.73:0/3938819372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c102ed0 msgr2=0x7fec2c109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 --2- 172.21.15.73:0/3938819372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c102ed0 0x7fec2c109760 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fec24009900 tx=0x7fec2402bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 -- 172.21.15.73:0/3938819372 shutdown_connections 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 --2- 172.21.15.73:0/3938819372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c102ed0 0x7fec2c109760 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 --2- 172.21.15.73:0/3938819372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 0x7fec2c102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 --2- 172.21.15.73:0/3938819372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c101320 0x7fec2c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 -- 172.21.15.73:0/3938819372 >> 172.21.15.73:0/3938819372 conn(0x7fec2c0fc890 msgr2=0x7fec2c0fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 -- 172.21.15.73:0/3938819372 shutdown_connections 2024-03-09T00:03:35.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.145+0000 7fec31522640 1 -- 172.21.15.73:0/3938819372 wait complete. 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 Processor -- start 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 -- start start 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 0x7fec2c197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c102ed0 0x7fec2c19eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c10cfb0 con 0x7fec2c101320 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c10ce30 con 0x7fec2c102ed0 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec31522640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec2c10d130 con 0x7fec2c102510 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec2affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec2affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33956/0 (socket says 172.21.15.73:33956) 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec2affd640 1 -- 172.21.15.73:0/2869982832 learned_addr learned my addr 172.21.15.73:0/2869982832 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec2b7fe640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c102ed0 0x7fec2c19eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.146+0000 7fec2a7fc640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 0x7fec2c197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:35.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 -- 172.21.15.73:0/2869982832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 msgr2=0x7fec2c197af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 0x7fec2c197af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 -- 172.21.15.73:0/2869982832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c102ed0 msgr2=0x7fec2c19eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c102ed0 0x7fec2c19eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 -- 172.21.15.73:0/2869982832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec2c19f1d0 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec2affd640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00ebc0 tx=0x7fec1c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c003b70 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec1c003d10 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec2c19f4c0 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec1c019980 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.147+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec2c19fa10 con 0x7fec2c101320 2024-03-09T00:03:35.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.148+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec2c108640 con 0x7fec2c101320 2024-03-09T00:03:35.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.149+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fec1c019b20 con 0x7fec2c101320 2024-03-09T00:03:35.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.150+0000 7fec13fff640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 0x7fec140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:35.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.150+0000 7fec2a7fc640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 0x7fec140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:35.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.150+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fec1c0ceda0 con 0x7fec2c101320 2024-03-09T00:03:35.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.150+0000 7fec2a7fc640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 0x7fec140a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec2c198ac0 tx=0x7fec2000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:35.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.154+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fec1c09c2e0 con 0x7fec2c101320 2024-03-09T00:03:35.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.367+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec2c02ce70 con 0x7fec2c101320 2024-03-09T00:03:35.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.368+0000 7fec13fff640 1 -- 172.21.15.73:0/2869982832 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fec1c09bbe0 con 0x7fec2c101320 2024-03-09T00:03:35.363 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:35.363 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-09T00:03:35.364 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 1 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 msgr2=0x7fec140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 0x7fec140a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fec2c198ac0 tx=0x7fec2000b410 comp rx=0 tx=0).stop 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 msgr2=0x7fec2c1975b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fec1c00ebc0 tx=0x7fec1c00c6a0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 shutdown_connections 2024-03-09T00:03:35.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec140a2800 0x7fec140a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec2c102ed0 0x7fec2c19eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec2c102510 0x7fec2c197af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 --2- 172.21.15.73:0/2869982832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec2c101320 0x7fec2c1975b0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 >> 172.21.15.73:0/2869982832 conn(0x7fec2c0fc890 msgr2=0x7fec2c105810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 shutdown_connections 2024-03-09T00:03:35.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:35.370+0000 7fec31522640 1 -- 172.21.15.73:0/2869982832 wait complete. 2024-03-09T00:03:35.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:35 smithi073 ceph-mon[32309]: Deploying daemon mds.l on smithi179 2024-03-09T00:03:35.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2004601906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:35.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3325893397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:35.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:35 smithi102 ceph-mon[36106]: Deploying daemon mds.l on smithi179 2024-03-09T00:03:35.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2004601906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:35.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3325893397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:35.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:35 smithi179 ceph-mon[36637]: Deploying daemon mds.l on smithi179 2024-03-09T00:03:35.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2004601906' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:03:35.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3325893397' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:03:36.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:36 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2869982832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:36.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:36 smithi179 ceph-mon[36637]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:36.582 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 systemd[1]: Starting Ceph mds.l for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:36.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:36 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2869982832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:36.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:36 smithi073 ceph-mon[32309]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:36.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:36 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2869982832' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:36.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:36 smithi102 ceph-mon[36106]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:36.887 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 podman[66643]: 2024-03-09 00:03:36.53376582 +0000 UTC m=+0.017534715 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:36.887 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 podman[66643]: 2024-03-09 00:03:36.638614229 +0000 UTC m=+0.122383103 container create 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-09T00:03:36.887 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 podman[66643]: 2024-03-09 00:03:36.866553464 +0000 UTC m=+0.350322346 container init 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-09T00:03:36.887 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 podman[66643]: 2024-03-09 00:03:36.86925867 +0000 UTC m=+0.353027546 container start 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-09T00:03:36.887 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 ceph-mds[66662]: -- 172.21.15.179:0/2272201621 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e6a9dc7a40 con 0x55e6a9d9b000 2024-03-09T00:03:37.331 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l[66658]: starting mds.l at 2024-03-09T00:03:37.332 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 bash[66643]: 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 2024-03-09T00:03:37.332 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:03:36 smithi179 systemd[1]: Started Ceph mds.l for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:37.377 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:37.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- 172.21.15.73:0/180512080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 msgr2=0x7f4900101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:37.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- 172.21.15.73:0/180512080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900101730 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f48f8009900 tx=0x7f48f802bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:37.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- 172.21.15.73:0/180512080 shutdown_connections 2024-03-09T00:03:37.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- 172.21.15.73:0/180512080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 0x7f4900109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.486 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- 172.21.15.73:0/180512080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 0x7f4900102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- 172.21.15.73:0/180512080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900101730 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- 172.21.15.73:0/180512080 >> 172.21.15.73:0/180512080 conn(0x7f49000fc8a0 msgr2=0x7f49000fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- 172.21.15.73:0/180512080 shutdown_connections 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- 172.21.15.73:0/180512080 wait complete. 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 Processor -- start 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- start start 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 0x7f4900197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 0x7f490019ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010cfa0 con 0x7f4900101310 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010ce20 con 0x7f4900102ec0 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f4907406640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f490010d120 con 0x7f4900102500 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f49051a2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:37.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f49051a2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33962/0 (socket says 172.21.15.73:33962) 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.489+0000 7f49051a2640 1 -- 172.21.15.73:0/3415506189 learned_addr learned my addr 172.21.15.73:0/3415506189 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49059a3640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 0x7f490019ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49049a1640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 0x7f4900197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 -- 172.21.15.73:0/3415506189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 msgr2=0x7f4900197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 0x7f4900197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 -- 172.21.15.73:0/3415506189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 msgr2=0x7f490019ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 0x7f490019ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 -- 172.21.15.73:0/3415506189 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490019f250 con 0x7f4900101310 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f49051a2640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f48f8000c00 tx=0x7f48f8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8041070 con 0x7f4900101310 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f48f80044d0 con 0x7f4900101310 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f490019f4e0 con 0x7f4900101310 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48f8034440 con 0x7f4900101310 2024-03-09T00:03:37.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.490+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f490019f9d0 con 0x7f4900101310 2024-03-09T00:03:37.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.491+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4900108630 con 0x7f4900101310 2024-03-09T00:03:37.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.492+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f48f8044070 con 0x7f4900101310 2024-03-09T00:03:37.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.493+0000 7f48f67fc640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 0x7f48ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:37.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.493+0000 7f49049a1640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 0x7f48ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:37.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.493+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f48f80f00b0 con 0x7f4900101310 2024-03-09T00:03:37.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.493+0000 7f49049a1640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 0x7f48ec0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4900198b40 tx=0x7f48f000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:37.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.497+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f48f80bcfa0 con 0x7f4900101310 2024-03-09T00:03:37.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.708+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f490002d090 con 0x7f4900101310 2024-03-09T00:03:37.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.709+0000 7f48f67fc640 1 -- 172.21.15.73:0/3415506189 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f48f80c1040 con 0x7f4900101310 2024-03-09T00:03:37.704 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:37.704 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-09T00:03:37.704 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 2 2024-03-09T00:03:37.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 msgr2=0x7f48ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:37.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 0x7f48ec0a4cc0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f4900198b40 tx=0x7f48f000b390 comp rx=0 tx=0).stop 2024-03-09T00:03:37.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 msgr2=0x7f4900197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:37.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f48f8000c00 tx=0x7f48f8005d50 comp rx=0 tx=0).stop 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 shutdown_connections 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f48ec0a2800 0x7f48ec0a4cc0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4900102ec0 0x7f490019ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4900102500 0x7f4900197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 --2- 172.21.15.73:0/3415506189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4900101310 0x7f4900197630 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 >> 172.21.15.73:0/3415506189 conn(0x7f49000fc8a0 msgr2=0x7f4900105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:37.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 shutdown_connections 2024-03-09T00:03:37.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:37.711+0000 7f4907406640 1 -- 172.21.15.73:0/3415506189 wait complete. 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.179:6832/3042063350,v1:172.21.15.179:6833/3042063350] up:boot 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: fsmap 1 up:standby 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:37.941 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:37.942 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:37.942 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:37.942 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: Deploying daemon mds.k on smithi102 2024-03-09T00:03:37.942 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:37 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3415506189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:38.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.179:6832/3042063350,v1:172.21.15.179:6833/3042063350] up:boot 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: fsmap 1 up:standby 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: Deploying daemon mds.k on smithi102 2024-03-09T00:03:38.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:37 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3415506189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.179:6832/3042063350,v1:172.21.15.179:6833/3042063350] up:boot 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: fsmap 1 up:standby 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: Deploying daemon mds.k on smithi102 2024-03-09T00:03:38.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:37 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3415506189' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:39 smithi179 ceph-mon[36637]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:39 smithi073 ceph-mon[32309]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:39.490 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:39 smithi102 ceph-mon[36106]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:39.491 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 systemd[1]: Starting Ceph mds.k for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:39.718 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:39.759 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 podman[65539]: 2024-03-09 00:03:39.398572294 +0000 UTC m=+0.017467848 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:39.760 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 podman[65539]: 2024-03-09 00:03:39.521160114 +0000 UTC m=+0.140055701 container create 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 -- 172.21.15.73:0/3438042244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102ee0 msgr2=0x7f7210109770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 --2- 172.21.15.73:0/3438042244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102ee0 0x7f7210109770 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7208009ca0 tx=0x7f720802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 -- 172.21.15.73:0/3438042244 shutdown_connections 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 --2- 172.21.15.73:0/3438042244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102ee0 0x7f7210109770 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 --2- 172.21.15.73:0/3438042244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102520 0x7f72101029a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 --2- 172.21.15.73:0/3438042244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 0x7f7210101750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 -- 172.21.15.73:0/3438042244 >> 172.21.15.73:0/3438042244 conn(0x7f72100fc8a0 msgr2=0x7f72100fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 -- 172.21.15.73:0/3438042244 shutdown_connections 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.843+0000 7f7216f69640 1 -- 172.21.15.73:0/3438042244 wait complete. 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 Processor -- start 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 -- start start 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 0x7f72101975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102ee0 0x7f721019eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721010cfc0 con 0x7f7210102520 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721010ce40 con 0x7f7210102ee0 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7216f69640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f721010d140 con 0x7f7210101330 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38430/0 (socket says 172.21.15.73:38430) 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 -- 172.21.15.73:0/1625724188 learned_addr learned my addr 172.21.15.73:0/1625724188 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7215506640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102ee0 0x7f721019eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f7214d05640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 0x7f72101975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 -- 172.21.15.73:0/1625724188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 msgr2=0x7f72101975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 0x7f72101975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 -- 172.21.15.73:0/1625724188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102ee0 msgr2=0x7f721019eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102ee0 0x7f721019eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 -- 172.21.15.73:0/1625724188 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f721019f1d0 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.844+0000 7f720ffff640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9f0 tx=0x7f71fc00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.845+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc00cd80 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.845+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f71fc010910 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.845+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f721019f4c0 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.845+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f71fc018990 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.845+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f721019fa60 con 0x7f7210102520 2024-03-09T00:03:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.846+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7210108650 con 0x7f7210102520 2024-03-09T00:03:39.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.847+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f71fc010ac0 con 0x7f7210102520 2024-03-09T00:03:39.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.847+0000 7f720dffb640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 0x7f72000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:39.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.848+0000 7f7214d05640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 0x7f72000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:39.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.848+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f71fc0ce920 con 0x7f7210102520 2024-03-09T00:03:39.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.848+0000 7f7214d05640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 0x7f72000a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7210102380 tx=0x7f7204005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:39.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:39.852+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f71fc09be60 con 0x7f7210102520 2024-03-09T00:03:40.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.070+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f721002d0c0 con 0x7f7210102520 2024-03-09T00:03:40.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.071+0000 7f720dffb640 1 -- 172.21.15.73:0/1625724188 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f71fc09b760 con 0x7f7210102520 2024-03-09T00:03:40.068 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:40.068 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-09T00:03:40.069 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 3 2024-03-09T00:03:40.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 msgr2=0x7f72000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:40.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 0x7f72000a4d40 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f7210102380 tx=0x7f7204005f90 comp rx=0 tx=0).stop 2024-03-09T00:03:40.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 msgr2=0x7f7210197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:40.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f71fc00e9f0 tx=0x7f71fc00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 shutdown_connections 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72000a2880 0x7f72000a4d40 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7210102ee0 0x7f721019eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7210102520 0x7f7210197af0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 --2- 172.21.15.73:0/1625724188 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7210101330 0x7f72101975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 >> 172.21.15.73:0/1625724188 conn(0x7f72100fc8a0 msgr2=0x7f7210105820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 shutdown_connections 2024-03-09T00:03:40.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:40.073+0000 7f7216f69640 1 -- 172.21.15.73:0/1625724188 wait complete. 2024-03-09T00:03:40.259 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 podman[65539]: 2024-03-09 00:03:39.799618571 +0000 UTC m=+0.418514120 container init 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-09T00:03:40.259 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 podman[65539]: 2024-03-09 00:03:39.802133429 +0000 UTC m=+0.421028975 container start 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0) 2024-03-09T00:03:40.259 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 ceph-mds[65558]: -- 172.21.15.102:0/1050372557 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b0a15f3a40 con 0x55b0a15c6c00 2024-03-09T00:03:40.259 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k[65554]: starting mds.k at 2024-03-09T00:03:40.260 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 bash[65539]: 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce 2024-03-09T00:03:40.260 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:03:39 smithi102 systemd[1]: Started Ceph mds.k for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: Deploying daemon mds.j on smithi073 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6832/2292586216,v1:172.21.15.102:6833/2292586216] up:boot 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: fsmap 2 up:standby 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:03:41.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:40 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1625724188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:41.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: Deploying daemon mds.j on smithi073 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6832/2292586216,v1:172.21.15.102:6833/2292586216] up:boot 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: fsmap 2 up:standby 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:03:41.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:40 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1625724188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:41.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: Deploying daemon mds.j on smithi073 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6832/2292586216,v1:172.21.15.102:6833/2292586216] up:boot 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: fsmap 2 up:standby 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:03:41.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:40 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1625724188' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:42.081 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:42.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 -- 172.21.15.73:0/3945657735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 msgr2=0x7f7804101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3945657735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f7804101880 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f77f4009900 tx=0x7f77f402bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 -- 172.21.15.73:0/3945657735 shutdown_connections 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3945657735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804101dc0 0x7f7804108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3945657735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f7804101880 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3945657735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804100210 0x7f7804100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.205+0000 7f780c3fa640 1 -- 172.21.15.73:0/3945657735 >> 172.21.15.73:0/3945657735 conn(0x7f78040fb780 msgr2=0x7f78040fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- 172.21.15.73:0/3945657735 shutdown_connections 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- 172.21.15.73:0/3945657735 wait complete. 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 Processor -- start 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- start start 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 0x7f7804068fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:42.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804101dc0 0x7f780419e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780410bef0 con 0x7f7804101400 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780410bd70 con 0x7f7804100210 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780c3fa640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f780410c070 con 0x7f7804101dc0 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f7809995640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f7809995640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38452/0 (socket says 172.21.15.73:38452) 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f7809995640 1 -- 172.21.15.73:0/3546795070 learned_addr learned my addr 172.21.15.73:0/3546795070 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780a196640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 0x7f7804068fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.206+0000 7f780a997640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804101dc0 0x7f780419e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 -- 172.21.15.73:0/3546795070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804101dc0 msgr2=0x7f780419e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804101dc0 0x7f780419e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 -- 172.21.15.73:0/3546795070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 msgr2=0x7f7804068fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 0x7f7804068fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 -- 172.21.15.73:0/3546795070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780419f260 con 0x7f7804101400 2024-03-09T00:03:42.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f780a196640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 0x7f7804068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f7809995640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f77f4005ab0 tx=0x7f77f402c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4043020 con 0x7f7804101400 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f77f40054e0 con 0x7f7804101400 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f780419f490 con 0x7f7804101400 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f77f4003ce0 con 0x7f7804101400 2024-03-09T00:03:42.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.207+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f780419f9e0 con 0x7f7804101400 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.208+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7804107530 con 0x7f7804101400 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.209+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f77f4046020 con 0x7f7804101400 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.209+0000 7f77fb7fe640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 0x7f77f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.210+0000 7f780a196640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 0x7f77f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.210+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f77f40f0690 con 0x7f7804101400 2024-03-09T00:03:42.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.210+0000 7f780a196640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 0x7f77f00a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7800005fd0 tx=0x7f780000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:42.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.214+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f77f404d020 con 0x7f7804101400 2024-03-09T00:03:42.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.470+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f780402ce70 con 0x7f7804101400 2024-03-09T00:03:42.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.471+0000 7f77fb7fe640 1 -- 172.21.15.73:0/3546795070 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7f77f40bdbd0 con 0x7f7804101400 2024-03-09T00:03:42.466 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:42.466 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-09T00:03:42.466 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 3 2024-03-09T00:03:42.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 msgr2=0x7f77f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:42.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 0x7f77f00a4cc0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f7800005fd0 tx=0x7f780000b390 comp rx=0 tx=0).stop 2024-03-09T00:03:42.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 msgr2=0x7f780419c330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f77f4005ab0 tx=0x7f77f402c6c0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 shutdown_connections 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f77f00a2800 0x7f77f00a4cc0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7804101dc0 0x7f780419e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7804101400 0x7f780419c330 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 --2- 172.21.15.73:0/3546795070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7804100210 0x7f7804068fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 >> 172.21.15.73:0/3546795070 conn(0x7f78040fb780 msgr2=0x7f7804104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 shutdown_connections 2024-03-09T00:03:42.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:42.473+0000 7f780c3fa640 1 -- 172.21.15.73:0/3546795070 wait complete. 2024-03-09T00:03:42.651 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 podman[73242]: 2024-03-09 00:03:42.581974442 +0000 UTC m=+0.119787228 container create 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-09T00:03:42.950 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 podman[73242]: 2024-03-09 00:03:42.810160326 +0000 UTC m=+0.347973086 container init 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:03:42.950 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 podman[73242]: 2024-03-09 00:03:42.812845104 +0000 UTC m=+0.350657863 container start 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:03:42.950 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 ceph-mds[73262]: -- 172.21.15.73:0/2840285934 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562886ab9a40 con 0x562886a8d000 2024-03-09T00:03:42.951 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j[73258]: starting mds.j at 2024-03-09T00:03:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:42 smithi102 ceph-mon[36106]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:42 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3546795070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:42 smithi179 ceph-mon[36637]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:42 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3546795070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:42 smithi073 ceph-mon[32309]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:42 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3546795070' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:43.402 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 bash[73242]: 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 2024-03-09T00:03:43.402 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:03:42 smithi073 systemd[1]: Started Ceph mds.j for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.73:6834/1664042644,v1:172.21.15.73:6835/1664042644] up:boot 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: fsmap 3 up:standby 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:44.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:44.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:43 smithi073 ceph-mon[32309]: Deploying daemon mds.d on smithi073 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.73:6834/1664042644,v1:172.21.15.73:6835/1664042644] up:boot 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: fsmap 3 up:standby 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:44.271 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:43 smithi102 ceph-mon[36106]: Deploying daemon mds.d on smithi073 2024-03-09T00:03:44.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.73:6834/1664042644,v1:172.21.15.73:6835/1664042644] up:boot 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: fsmap 3 up:standby 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:43 smithi179 ceph-mon[36637]: Deploying daemon mds.d on smithi073 2024-03-09T00:03:44.481 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- 172.21.15.73:0/3679929909 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4100210 msgr2=0x7f1ec4100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- 172.21.15.73:0/3679929909 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4100210 0x7f1ec4100630 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4009900 tx=0x7f1eb402bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- 172.21.15.73:0/3679929909 shutdown_connections 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- 172.21.15.73:0/3679929909 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4101dc0 0x7f1ec4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- 172.21.15.73:0/3679929909 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101400 0x7f1ec4101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- 172.21.15.73:0/3679929909 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4100210 0x7f1ec4100630 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- 172.21.15.73:0/3679929909 >> 172.21.15.73:0/3679929909 conn(0x7f1ec40fb780 msgr2=0x7f1ec40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- 172.21.15.73:0/3679929909 shutdown_connections 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- 172.21.15.73:0/3679929909 wait complete. 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 Processor -- start 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- start start 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4100210 0x7f1ec4197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101dc0 0x7f1ec419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec410bea0 con 0x7f1ec4101400 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec410bd20 con 0x7f1ec4101dc0 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ecae61640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ec410c020 con 0x7f1ec4100210 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ec3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ec3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38492/0 (socket says 172.21.15.73:38492) 2024-03-09T00:03:44.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.595+0000 7f1ec3fff640 1 -- 172.21.15.73:0/2745060362 learned_addr learned my addr 172.21.15.73:0/2745060362 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec8bfd640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4100210 0x7f1ec4197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 -- 172.21.15.73:0/2745060362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4100210 msgr2=0x7f1ec4197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4100210 0x7f1ec4197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 -- 172.21.15.73:0/2745060362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101dc0 msgr2=0x7f1ec419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec93fe640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101dc0 0x7f1ec419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101dc0 0x7f1ec419ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 -- 172.21.15.73:0/2745060362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ec419f240 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec3fff640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d950 tx=0x7f1eb800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb8017070 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ec419f530 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1eb8009c30 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eb800f6b0 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.596+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ec419fa80 con 0x7f1ec4101400 2024-03-09T00:03:44.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.597+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ec419fce0 con 0x7f1ec4101400 2024-03-09T00:03:44.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.598+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1eb80109d0 con 0x7f1ec4101400 2024-03-09T00:03:44.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.599+0000 7f1ec1ffb640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 0x7f1eac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:44.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.599+0000 7f1ec8bfd640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 0x7f1eac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:44.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.599+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1eb80ce270 con 0x7f1ec4101400 2024-03-09T00:03:44.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.599+0000 7f1ec8bfd640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 0x7f1eac0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4000c00 tx=0x7f1eb4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:44.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.603+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1eb8009de0 con 0x7f1ec4101400 2024-03-09T00:03:44.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.821+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ec419feb0 con 0x7f1ec4101400 2024-03-09T00:03:44.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.821+0000 7f1ec1ffb640 1 -- 172.21.15.73:0/2745060362 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2913 (secure 0 0 0) 0x7f1eb809b7b0 con 0x7f1ec4101400 2024-03-09T00:03:44.817 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:44.817 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-09T00:03:44.817 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 4 2024-03-09T00:03:44.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 msgr2=0x7f1eac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:44.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 0x7f1eac0a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f1eb4000c00 tx=0x7f1eb4005fd0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 msgr2=0x7f1ec4197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:44.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f1eb800d950 tx=0x7f1eb800de10 comp rx=0 tx=0).stop 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 shutdown_connections 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1eac0a2800 0x7f1eac0a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ec4101dc0 0x7f1ec419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ec4101400 0x7f1ec4197b60 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 --2- 172.21.15.73:0/2745060362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ec4100210 0x7f1ec4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 >> 172.21.15.73:0/2745060362 conn(0x7f1ec40fb780 msgr2=0x7f1ec40fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 shutdown_connections 2024-03-09T00:03:44.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:44.823+0000 7f1ecae61640 1 -- 172.21.15.73:0/2745060362 wait complete. 2024-03-09T00:03:45.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:45 smithi179 ceph-mon[36637]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:45.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:45 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2745060362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:45.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:45 smithi073 ceph-mon[32309]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:45.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:45 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2745060362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:45.402 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 systemd[1]: Starting Ceph mds.d for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:45.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:45 smithi102 ceph-mon[36106]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:45.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:45 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2745060362' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:45.737 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 podman[73591]: 2024-03-09 00:03:45.355815706 +0000 UTC m=+0.018573269 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:45.737 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 podman[73591]: 2024-03-09 00:03:45.465229204 +0000 UTC m=+0.127986800 container create c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD) 2024-03-09T00:03:46.151 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 podman[73591]: 2024-03-09 00:03:45.743321982 +0000 UTC m=+0.406079535 container init c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307) 2024-03-09T00:03:46.151 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 podman[73591]: 2024-03-09 00:03:45.745972031 +0000 UTC m=+0.408729595 container start c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-09T00:03:46.152 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 ceph-mds[73610]: -- 172.21.15.73:0/62314159 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560db33a7a40 con 0x560db337ac00 2024-03-09T00:03:46.152 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d[73606]: starting mds.d at 2024-03-09T00:03:46.152 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 bash[73591]: c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee 2024-03-09T00:03:46.152 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:03:45 smithi073 systemd[1]: Started Ceph mds.d for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:46.829 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:46.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 -- 172.21.15.73:0/771815359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 msgr2=0x7f2d581006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 --2- 172.21.15.73:0/771815359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d581006b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f2d54009900 tx=0x7f2d5402bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 -- 172.21.15.73:0/771815359 shutdown_connections 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 --2- 172.21.15.73:0/771815359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101e40 0x7f2d581086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 --2- 172.21.15.73:0/771815359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101480 0x7f2d58101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 --2- 172.21.15.73:0/771815359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d581006b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 -- 172.21.15.73:0/771815359 >> 172.21.15.73:0/771815359 conn(0x7f2d580fb820 msgr2=0x7f2d580fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.952+0000 7f2d604f1640 1 -- 172.21.15.73:0/771815359 shutdown_connections 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 -- 172.21.15.73:0/771815359 wait complete. 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 Processor -- start 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 -- start start 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101480 0x7f2d5819c3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:46.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101e40 0x7f2d5819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5810be80 con 0x7f2d58100290 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5810bd00 con 0x7f2d58101e40 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d604f1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d5810c000 con 0x7f2d58101480 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d5e28d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d5e28d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38540/0 (socket says 172.21.15.73:38540) 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d5e28d640 1 -- 172.21.15.73:0/3127202236 learned_addr learned my addr 172.21.15.73:0/3127202236 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d5da8c640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101480 0x7f2d5819c3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.953+0000 7f2d5ea8e640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101e40 0x7f2d5819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 -- 172.21.15.73:0/3127202236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101480 msgr2=0x7f2d5819c3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101480 0x7f2d5819c3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 -- 172.21.15.73:0/3127202236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101e40 msgr2=0x7f2d5819e9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101e40 0x7f2d5819e9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 -- 172.21.15.73:0/3127202236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d5819f2b0 con 0x7f2d58100290 2024-03-09T00:03:46.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d5e28d640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2d54005ab0 tx=0x7f2d54005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:46.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d54041070 con 0x7f2d58100290 2024-03-09T00:03:46.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d540054e0 con 0x7f2d58100290 2024-03-09T00:03:46.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d5819f540 con 0x7f2d58100290 2024-03-09T00:03:46.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d54034440 con 0x7f2d58100290 2024-03-09T00:03:46.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.954+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d5819fa30 con 0x7f2d58100290 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.955+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d581076d0 con 0x7f2d58100290 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.956+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2d54044070 con 0x7f2d58100290 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.957+0000 7f2d4f7fe640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 0x7f2d440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.957+0000 7f2d5da8c640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 0x7f2d440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.957+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2d540f03b0 con 0x7f2d58100290 2024-03-09T00:03:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.958+0000 7f2d5da8c640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 0x7f2d440a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806a200 tx=0x7f2d4800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:46.961+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2d540bd8f0 con 0x7f2d58100290 2024-03-09T00:03:47.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: Deploying daemon mds.e on smithi102 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.73:6836/3028984722,v1:172.21.15.73:6837/3028984722] up:boot 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: fsmap 4 up:standby 2024-03-09T00:03:47.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:46 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:03:47.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.172+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d5802d0c0 con 0x7f2d58100290 2024-03-09T00:03:47.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.173+0000 7f2d4f7fe640 1 -- 172.21.15.73:0/3127202236 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3704 (secure 0 0 0) 0x7f2d540bd1f0 con 0x7f2d58100290 2024-03-09T00:03:47.168 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:47.168 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-09T00:03:47.168 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 5 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 msgr2=0x7f2d440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 0x7f2d440a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f2d5806a200 tx=0x7f2d4800b410 comp rx=0 tx=0).stop 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 msgr2=0x7f2d5806a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f2d54005ab0 tx=0x7f2d54005dc0 comp rx=0 tx=0).stop 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 shutdown_connections 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2d440a2800 0x7f2d440a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d58101e40 0x7f2d5819e9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d58101480 0x7f2d5819c3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:47.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 --2- 172.21.15.73:0/3127202236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d58100290 0x7f2d5806a490 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:47.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 >> 172.21.15.73:0/3127202236 conn(0x7f2d580fb820 msgr2=0x7f2d581048a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:47.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 shutdown_connections 2024-03-09T00:03:47.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:47.175+0000 7f2d604f1640 1 -- 172.21.15.73:0/3127202236 wait complete. 2024-03-09T00:03:47.255 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.255 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.255 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:47.255 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: Deploying daemon mds.e on smithi102 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.73:6836/3028984722,v1:172.21.15.73:6837/3028984722] up:boot 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: fsmap 4 up:standby 2024-03-09T00:03:47.256 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:46 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: Deploying daemon mds.e on smithi102 2024-03-09T00:03:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.73:6836/3028984722,v1:172.21.15.73:6837/3028984722] up:boot 2024-03-09T00:03:47.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: fsmap 4 up:standby 2024-03-09T00:03:47.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:46 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:03:48.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:47 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3127202236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:48.156 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:47 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3127202236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:48.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:47 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3127202236' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:48.509 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 podman[65864]: 2024-03-09 00:03:48.411894093 +0000 UTC m=+0.149284870 container create fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.build-date=20240307) 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 podman[65864]: 2024-03-09 00:03:48.656735233 +0000 UTC m=+0.394126018 container init fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 podman[65864]: 2024-03-09 00:03:48.659428418 +0000 UTC m=+0.396819204 container start fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mds[65884]: -- 172.21.15.102:0/2387334936 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bae249da40 con 0x55bae2470800 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e[65880]: starting mds.e at 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 bash[65864]: fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 2024-03-09T00:03:48.893 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:03:48 smithi102 systemd[1]: Started Ceph mds.e for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:49.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:49.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:49.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:49.181 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:49.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:49.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:49.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 -- 172.21.15.73:0/2029645985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4100230 msgr2=0x7fcec4100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:49.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 --2- 172.21.15.73:0/2029645985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4100230 0x7fcec4100650 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fcec0009900 tx=0x7fcec002bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:49.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 -- 172.21.15.73:0/2029645985 shutdown_connections 2024-03-09T00:03:49.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 --2- 172.21.15.73:0/2029645985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101de0 0x7fcec4108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 --2- 172.21.15.73:0/2029645985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4101420 0x7fcec41018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 --2- 172.21.15.73:0/2029645985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4100230 0x7fcec4100650 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 -- 172.21.15.73:0/2029645985 >> 172.21.15.73:0/2029645985 conn(0x7fcec40fb7a0 msgr2=0x7fcec40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:49.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 -- 172.21.15.73:0/2029645985 shutdown_connections 2024-03-09T00:03:49.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.299+0000 7fcecb827640 1 -- 172.21.15.73:0/2029645985 wait complete. 2024-03-09T00:03:49.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 Processor -- start 2024-03-09T00:03:49.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 -- start start 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 0x7fcec4196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101420 0x7fcec4196a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec410bf10 con 0x7fcec4101de0 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec410bd90 con 0x7fcec4100230 2024-03-09T00:03:49.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcecb827640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcec410c090 con 0x7fcec4101420 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42636/0 (socket says 172.21.15.73:42636) 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 -- 172.21.15.73:0/2014868822 learned_addr learned my addr 172.21.15.73:0/2014868822 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec8dc2640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101420 0x7fcec4196a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec95c3640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 0x7fcec4196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:49.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 -- 172.21.15.73:0/2014868822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101420 msgr2=0x7fcec4196a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:49.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101420 0x7fcec4196a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 -- 172.21.15.73:0/2014868822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 msgr2=0x7fcec4196540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:49.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 0x7fcec4196540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 -- 172.21.15.73:0/2014868822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcec4069920 con 0x7fcec4101de0 2024-03-09T00:03:49.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec95c3640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 0x7fcec4196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:03:49.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.300+0000 7fcec9dc4640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00eb10 tx=0x7fcebc00c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:49.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.301+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc004b20 con 0x7fcec4101de0 2024-03-09T00:03:49.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.301+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcebc004cc0 con 0x7fcec4101de0 2024-03-09T00:03:49.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.301+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcec4069c10 con 0x7fcec4101de0 2024-03-09T00:03:49.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.301+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcebc010760 con 0x7fcec4101de0 2024-03-09T00:03:49.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.301+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcec406a1b0 con 0x7fcec4101de0 2024-03-09T00:03:49.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.302+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcec4107550 con 0x7fcec4101de0 2024-03-09T00:03:49.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.303+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fcebc005020 con 0x7fcec4101de0 2024-03-09T00:03:49.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.303+0000 7fceba7fc640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 0x7fceb00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:49.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.304+0000 7fcec95c3640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 0x7fceb00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:49.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.304+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fcebc0cecc0 con 0x7fcec4101de0 2024-03-09T00:03:49.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.304+0000 7fcec95c3640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 0x7fceb00a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcec00096c0 tx=0x7fcec002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:49.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.308+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcebc09c200 con 0x7fcec4101de0 2024-03-09T00:03:49.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:49.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:49.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:49.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:49.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:49.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.525+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcec406a470 con 0x7fcec4101de0 2024-03-09T00:03:49.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.525+0000 7fceba7fc640 1 -- 172.21.15.73:0/2014868822 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4495 (secure 0 0 0) 0x7fcebc09bb00 con 0x7fcec4101de0 2024-03-09T00:03:49.521 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:49.521 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-09T00:03:49.521 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 6 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 msgr2=0x7fceb00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 0x7fceb00a4d40 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fcec00096c0 tx=0x7fcec002df90 comp rx=0 tx=0).stop 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 msgr2=0x7fcec419daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00eb10 tx=0x7fcebc00c2d0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 shutdown_connections 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fceb00a2880 0x7fceb00a4d40 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcec4101de0 0x7fcec419daf0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcec4101420 0x7fcec4196a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 --2- 172.21.15.73:0/2014868822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcec4100230 0x7fcec4196540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:49.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.527+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 >> 172.21.15.73:0/2014868822 conn(0x7fcec40fb7a0 msgr2=0x7fcec4104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:49.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.528+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 shutdown_connections 2024-03-09T00:03:49.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:49.528+0000 7fcecb827640 1 -- 172.21.15.73:0/2014868822 wait complete. 2024-03-09T00:03:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:49 smithi179 ceph-mon[36637]: Deploying daemon mds.f on smithi179 2024-03-09T00:03:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:49 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6834/364340358,v1:172.21.15.102:6835/364340358] up:boot 2024-03-09T00:03:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:49 smithi179 ceph-mon[36637]: fsmap 5 up:standby 2024-03-09T00:03:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:49 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:03:50.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:49 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2014868822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:50.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:49 smithi073 ceph-mon[32309]: Deploying daemon mds.f on smithi179 2024-03-09T00:03:50.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:49 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6834/364340358,v1:172.21.15.102:6835/364340358] up:boot 2024-03-09T00:03:50.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:49 smithi073 ceph-mon[32309]: fsmap 5 up:standby 2024-03-09T00:03:50.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:49 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:03:50.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:49 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2014868822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:49 smithi102 ceph-mon[36106]: Deploying daemon mds.f on smithi179 2024-03-09T00:03:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:49 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6834/364340358,v1:172.21.15.102:6835/364340358] up:boot 2024-03-09T00:03:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:49 smithi102 ceph-mon[36106]: fsmap 5 up:standby 2024-03-09T00:03:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:49 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:03:50.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:49 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2014868822' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:51.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:50 smithi179 ceph-mon[36637]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:51.082 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:50 smithi179 systemd[1]: Starting Ceph mds.f for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:50 smithi073 ceph-mon[32309]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:51.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:50 smithi102 ceph-mon[36106]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:51.468 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 podman[66964]: 2024-03-09 00:03:51.10504884 +0000 UTC m=+0.017119135 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:51.469 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 podman[66964]: 2024-03-09 00:03:51.229179528 +0000 UTC m=+0.141249875 container create 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-09T00:03:51.469 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 podman[66964]: 2024-03-09 00:03:51.474469992 +0000 UTC m=+0.386540285 container init 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-09T00:03:51.534 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 -- 172.21.15.73:0/985523191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00101e10 msgr2=0x7f3d00104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 --2- 172.21.15.73:0/985523191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00101e10 0x7f3d00104200 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0009ca0 tx=0x7f3cf002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 -- 172.21.15.73:0/985523191 shutdown_connections 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 --2- 172.21.15.73:0/985523191 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00107070 0x7f3d00109460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 --2- 172.21.15.73:0/985523191 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 0x7f3d00106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 --2- 172.21.15.73:0/985523191 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00101e10 0x7f3d00104200 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 -- 172.21.15.73:0/985523191 >> 172.21.15.73:0/985523191 conn(0x7f3d000fb780 msgr2=0x7f3d000fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 -- 172.21.15.73:0/985523191 shutdown_connections 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.655+0000 7f3d05412640 1 -- 172.21.15.73:0/985523191 wait complete. 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 Processor -- start 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 -- start start 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00101e10 0x7f3d00199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:51.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 0x7f3d00199d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:51.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:51.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0010cfa0 con 0x7f3d00107070 2024-03-09T00:03:51.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0010ce20 con 0x7f3d00104740 2024-03-09T00:03:51.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3d05412640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d0010d120 con 0x7f3d00101e10 2024-03-09T00:03:51.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3cff7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42654/0 (socket says 172.21.15.73:42654) 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.656+0000 7f3cff7fe640 1 -- 172.21.15.73:0/3147747972 learned_addr learned my addr 172.21.15.73:0/3147747972 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cfeffd640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00101e10 0x7f3d00199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cfe7fc640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 0x7f3d00199d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 -- 172.21.15.73:0/3147747972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00101e10 msgr2=0x7f3d00199830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00101e10 0x7f3d00199830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 -- 172.21.15.73:0/3147747972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 msgr2=0x7f3d00199d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:51.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 0x7f3d00199d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 -- 172.21.15.73:0/3147747972 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d001a1450 con 0x7f3d00107070 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3cff7fe640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800e970 tx=0x7f3cf800ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf800ccc0 con 0x7f3d00107070 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d001a1740 con 0x7f3d00107070 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3cf800ce60 con 0x7f3d00107070 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3cf8010450 con 0x7f3d00107070 2024-03-09T00:03:51.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.657+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d001a1ce0 con 0x7f3d00107070 2024-03-09T00:03:51.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.658+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d00108b60 con 0x7f3d00107070 2024-03-09T00:03:51.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.660+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3cf8005020 con 0x7f3d00107070 2024-03-09T00:03:51.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.661+0000 7f3ce7fff640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 0x7f3ce80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:51.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.661+0000 7f3cfeffd640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 0x7f3ce80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:51.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.661+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3cf80ce9e0 con 0x7f3d00107070 2024-03-09T00:03:51.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.661+0000 7f3cfeffd640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 0x7f3ce80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006010 tx=0x7f3cf0032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:51.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.665+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3cf809bf20 con 0x7f3d00107070 2024-03-09T00:03:51.831 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 podman[66964]: 2024-03-09 00:03:51.477343871 +0000 UTC m=+0.389414154 container start 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-09T00:03:51.832 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 ceph-mds[66983]: -- 172.21.15.179:0/2483701804 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55f17cf55a40 con 0x55f17cf28800 2024-03-09T00:03:51.832 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f[66979]: starting mds.f at 2024-03-09T00:03:51.832 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 bash[66964]: 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c 2024-03-09T00:03:51.832 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:03:51 smithi179 systemd[1]: Started Ceph mds.f for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:51.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.883+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3d0002d020 con 0x7f3d00107070 2024-03-09T00:03:51.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.883+0000 7f3ce7fff640 1 -- 172.21.15.73:0/3147747972 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4495 (secure 0 0 0) 0x7f3cf809b820 con 0x7f3d00107070 2024-03-09T00:03:51.879 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:51.879 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-09T00:03:51.879 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 6 2024-03-09T00:03:51.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 msgr2=0x7f3ce80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 0x7f3ce80a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f3cf0006010 tx=0x7f3cf0032590 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 msgr2=0x7f3d001a0de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3cf800e970 tx=0x7f3cf800ed70 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 shutdown_connections 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ce80a2800 0x7f3ce80a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d00107070 0x7f3d001a0de0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d00104740 0x7f3d00199d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 --2- 172.21.15.73:0/3147747972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d00101e10 0x7f3d00199830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.885+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 >> 172.21.15.73:0/3147747972 conn(0x7f3d000fb780 msgr2=0x7f3d00101540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.886+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 shutdown_connections 2024-03-09T00:03:51.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:51.886+0000 7f3d05412640 1 -- 172.21.15.73:0/3147747972 wait complete. 2024-03-09T00:03:52.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: Deploying daemon mds.c on smithi179 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3147747972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.179:6834/2774016402,v1:172.21.15.179:6835/2774016402] up:boot 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: fsmap 6 up:standby 2024-03-09T00:03:52.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:52 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:03:52.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: Deploying daemon mds.c on smithi179 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3147747972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.179:6834/2774016402,v1:172.21.15.179:6835/2774016402] up:boot 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: fsmap 6 up:standby 2024-03-09T00:03:52.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:52 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: Deploying daemon mds.c on smithi179 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3147747972' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:53.010 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.179:6834/2774016402,v1:172.21.15.179:6835/2774016402] up:boot 2024-03-09T00:03:53.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: fsmap 6 up:standby 2024-03-09T00:03:53.011 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:52 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:03:53.893 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:54.001 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:53 smithi179 systemd[1]: Starting Ceph mds.c for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 -- 172.21.15.73:0/3129277599 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310107090 msgr2=0x7f5310109480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 --2- 172.21.15.73:0/3129277599 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310107090 0x7f5310109480 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f5308009900 tx=0x7f530802bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 -- 172.21.15.73:0/3129277599 shutdown_connections 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 --2- 172.21.15.73:0/3129277599 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310107090 0x7f5310109480 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 --2- 172.21.15.73:0/3129277599 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310104760 0x7f5310106b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 --2- 172.21.15.73:0/3129277599 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310101e30 0x7f5310104220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 -- 172.21.15.73:0/3129277599 >> 172.21.15.73:0/3129277599 conn(0x7f53100fb780 msgr2=0x7f53100fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 -- 172.21.15.73:0/3129277599 shutdown_connections 2024-03-09T00:03:54.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 -- 172.21.15.73:0/3129277599 wait complete. 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.014+0000 7f5314e52640 1 Processor -- start 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 -- start start 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 0x7f5310197540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310107090 0x7f531019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531010cfc0 con 0x7f5310104760 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531010ce40 con 0x7f5310101e30 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f531010d140 con 0x7f5310107090 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:54.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42680/0 (socket says 172.21.15.73:42680) 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 -- 172.21.15.73:0/2498122679 learned_addr learned my addr 172.21.15.73:0/2498122679 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530ed9d640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310107090 0x7f531019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530e59c640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 0x7f5310197540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 -- 172.21.15.73:0/2498122679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310107090 msgr2=0x7f531019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310107090 0x7f531019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 -- 172.21.15.73:0/2498122679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 msgr2=0x7f5310197540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 0x7f5310197540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 -- 172.21.15.73:0/2498122679 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f531019f160 con 0x7f5310104760 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530e59c640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 0x7f5310197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f530dd9b640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f530400c970 tx=0x7f530400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5304014070 con 0x7f5310104760 2024-03-09T00:03:54.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5304004440 con 0x7f5310104760 2024-03-09T00:03:54.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f531019f450 con 0x7f5310104760 2024-03-09T00:03:54.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.015+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f530400f450 con 0x7f5310104760 2024-03-09T00:03:54.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.016+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f531019f9a0 con 0x7f5310104760 2024-03-09T00:03:54.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.017+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53100fd270 con 0x7f5310104760 2024-03-09T00:03:54.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.017+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5304017070 con 0x7f5310104760 2024-03-09T00:03:54.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.018+0000 7f52ff7fe640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 0x7f52f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:54.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.018+0000 7f530e59c640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 0x7f52f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:54.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.018+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f53040ce4f0 con 0x7f5310104760 2024-03-09T00:03:54.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.018+0000 7f530e59c640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 0x7f52f80a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5300005fd0 tx=0x7f5300009520 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:54.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.022+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f530401e020 con 0x7f5310104760 2024-03-09T00:03:54.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.235+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f531002d040 con 0x7f5310104760 2024-03-09T00:03:54.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.235+0000 7f52ff7fe640 1 -- 172.21.15.73:0/2498122679 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5289 (secure 0 0 0) 0x7f530409ba30 con 0x7f5310104760 2024-03-09T00:03:54.231 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:54.231 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-09T00:03:54.231 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 7 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 msgr2=0x7f52f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 0x7f52f80a4cc0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5300005fd0 tx=0x7f5300009520 comp rx=0 tx=0).stop 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 msgr2=0x7f5310197a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f530400c970 tx=0x7f530400ce30 comp rx=0 tx=0).stop 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 shutdown_connections 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f52f80a2800 0x7f52f80a4cc0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5310107090 0x7f531019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5310104760 0x7f5310197a80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 --2- 172.21.15.73:0/2498122679 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5310101e30 0x7f5310197540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:54.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 >> 172.21.15.73:0/2498122679 conn(0x7f53100fb780 msgr2=0x7f5310102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:54.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 shutdown_connections 2024-03-09T00:03:54.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:54.237+0000 7f5314e52640 1 -- 172.21.15.73:0/2498122679 wait complete. 2024-03-09T00:03:54.301 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 podman[67270]: 2024-03-09 00:03:53.90985708 +0000 UTC m=+0.017320048 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:54.302 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 podman[67270]: 2024-03-09 00:03:54.037418481 +0000 UTC m=+0.144881458 container create 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-09T00:03:54.302 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 podman[67270]: 2024-03-09 00:03:54.307195047 +0000 UTC m=+0.414658004 container init 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-09T00:03:54.582 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 podman[67270]: 2024-03-09 00:03:54.309913757 +0000 UTC m=+0.417376716 container start 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:03:54.582 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mds[67289]: -- 172.21.15.179:0/1055640354 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5587803afa40 con 0x558780382c00 2024-03-09T00:03:54.582 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c[67285]: starting mds.c at 2024-03-09T00:03:54.582 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 bash[67270]: 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f 2024-03-09T00:03:54.583 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:03:54 smithi179 systemd[1]: Started Ceph mds.c for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2498122679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:55.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:55.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:54 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:55.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2498122679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:54 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2498122679' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:55.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:54 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:56.244 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:56.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:55 smithi102 ceph-mon[36106]: Deploying daemon mds.h on smithi102 2024-03-09T00:03:56.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:55 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.179:6836/2684774857,v1:172.21.15.179:6837/2684774857] up:boot 2024-03-09T00:03:56.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:55 smithi102 ceph-mon[36106]: fsmap 7 up:standby 2024-03-09T00:03:56.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:55 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:03:56.273 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:55 smithi073 ceph-mon[32309]: Deploying daemon mds.h on smithi102 2024-03-09T00:03:56.273 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:55 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.179:6836/2684774857,v1:172.21.15.179:6837/2684774857] up:boot 2024-03-09T00:03:56.273 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:55 smithi073 ceph-mon[32309]: fsmap 7 up:standby 2024-03-09T00:03:56.273 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:55 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:03:56.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:55 smithi179 ceph-mon[36637]: Deploying daemon mds.h on smithi102 2024-03-09T00:03:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:55 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.179:6836/2684774857,v1:172.21.15.179:6837/2684774857] up:boot 2024-03-09T00:03:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:55 smithi179 ceph-mon[36637]: fsmap 7 up:standby 2024-03-09T00:03:56.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:55 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 -- 172.21.15.73:0/2803860350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 msgr2=0x7f36e00ff810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 --2- 172.21.15.73:0/2803860350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e00ff810 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f36c8009900 tx=0x7f36c802bc30 comp rx=0 tx=0).stop 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 -- 172.21.15.73:0/2803860350 shutdown_connections 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 --2- 172.21.15.73:0/2803860350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 0x7f36e0109900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 --2- 172.21.15.73:0/2803860350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e00ff810 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 --2- 172.21.15.73:0/2803860350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 0x7f36e0102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 -- 172.21.15.73:0/2803860350 >> 172.21.15.73:0/2803860350 conn(0x7f36e00faf10 msgr2=0x7f36e00fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 -- 172.21.15.73:0/2803860350 shutdown_connections 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.363+0000 7f36e4881640 1 -- 172.21.15.73:0/2803860350 wait complete. 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 Processor -- start 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 -- start start 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 0x7f36e0197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 0x7f36e019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010d000 con 0x7f36e00ff390 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010ce80 con 0x7f36e0102920 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36e4881640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36e010d180 con 0x7f36e00ffd50 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36de59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36de59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42694/0 (socket says 172.21.15.73:42694) 2024-03-09T00:03:56.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.364+0000 7f36de59c640 1 -- 172.21.15.73:0/3903156327 learned_addr learned my addr 172.21.15.73:0/3903156327 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36ded9d640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 0x7f36e019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36d7fff640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 0x7f36e0197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 -- 172.21.15.73:0/3903156327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 msgr2=0x7f36e0197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 0x7f36e0197b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 -- 172.21.15.73:0/3903156327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 msgr2=0x7f36e019ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 0x7f36e019ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 -- 172.21.15.73:0/3903156327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36e019f260 con 0x7f36e00ff390 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36de59c640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36d000e9c0 tx=0x7f36d000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d000cd80 con 0x7f36e00ff390 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36e019f550 con 0x7f36e00ff390 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f36d00035f0 con 0x7f36e00ff390 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f36d0010690 con 0x7f36e00ff390 2024-03-09T00:03:56.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.365+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36e019faa0 con 0x7f36e00ff390 2024-03-09T00:03:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.366+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36e01087e0 con 0x7f36e00ff390 2024-03-09T00:03:56.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.367+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f36d00037a0 con 0x7f36e00ff390 2024-03-09T00:03:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.368+0000 7f36d77fe640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 0x7f36cc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.368+0000 7f36d7fff640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 0x7f36cc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.368+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f36d00cebb0 con 0x7f36e00ff390 2024-03-09T00:03:56.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.368+0000 7f36d7fff640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 0x7f36cc0a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f36c8036040 tx=0x7f36c8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:56.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.372+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f36d009c180 con 0x7f36e00ff390 2024-03-09T00:03:56.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.587+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f36e002ce70 con 0x7f36e00ff390 2024-03-09T00:03:56.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.587+0000 7f36d77fe640 1 -- 172.21.15.73:0/3903156327 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6083 (secure 0 0 0) 0x7f36d009ba80 con 0x7f36e00ff390 2024-03-09T00:03:56.583 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:56.583 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-09T00:03:56.584 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 8 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 msgr2=0x7f36cc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 0x7f36cc0a4cc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f36c8036040 tx=0x7f36c8005fd0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 msgr2=0x7f36e0197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f36d000e9c0 tx=0x7f36d000ee80 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 shutdown_connections 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f36cc0a2800 0x7f36cc0a4cc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36e0102920 0x7f36e019ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36e00ffd50 0x7f36e0197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 --2- 172.21.15.73:0/3903156327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36e00ff390 0x7f36e0197640 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:56.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 >> 172.21.15.73:0/3903156327 conn(0x7f36e00faf10 msgr2=0x7f36e01059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:56.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 shutdown_connections 2024-03-09T00:03:56.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:56.590+0000 7f36e4881640 1 -- 172.21.15.73:0/3903156327 wait complete. 2024-03-09T00:03:56.972 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:56 smithi102 systemd[1]: Starting Ceph mds.h for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:57.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:56 smithi179 ceph-mon[36637]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:57.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:56 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3903156327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:57.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:56 smithi102 ceph-mon[36106]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:57.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:56 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3903156327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:57.259 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:56 smithi102 podman[66172]: 2024-03-09 00:03:56.896907713 +0000 UTC m=+0.017910399 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:03:57.259 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 podman[66172]: 2024-03-09 00:03:57.035664466 +0000 UTC m=+0.156667133 container create 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-09T00:03:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:56 smithi073 ceph-mon[32309]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:56 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3903156327' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:57.759 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 podman[66172]: 2024-03-09 00:03:57.288569528 +0000 UTC m=+0.409572209 container init 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240307, GIT_CLEAN=True) 2024-03-09T00:03:57.759 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 podman[66172]: 2024-03-09 00:03:57.291256526 +0000 UTC m=+0.412259204 container start 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-09T00:03:57.759 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 ceph-mds[66192]: -- 172.21.15.102:0/3078283132 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56521b15da40 con 0x56521b130c00 2024-03-09T00:03:57.759 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h[66188]: starting mds.h at 2024-03-09T00:03:57.759 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 bash[66172]: 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 2024-03-09T00:03:57.760 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:03:57 smithi102 systemd[1]: Started Ceph mds.h for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:03:58.597 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:03:58.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 -- 172.21.15.73:0/2210294037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 msgr2=0x7f135c101cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 --2- 172.21.15.73:0/2210294037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c101cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f134c009900 tx=0x7f134c02bc80 comp rx=0 tx=0).stop 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 -- 172.21.15.73:0/2210294037 shutdown_connections 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 --2- 172.21.15.73:0/2210294037 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100e20 0x7f135c1086c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 --2- 172.21.15.73:0/2210294037 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100460 0x7f135c1008e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 --2- 172.21.15.73:0/2210294037 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c101cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 -- 172.21.15.73:0/2210294037 >> 172.21.15.73:0/2210294037 conn(0x7f135c0faf10 msgr2=0x7f135c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 -- 172.21.15.73:0/2210294037 shutdown_connections 2024-03-09T00:03:58.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.705+0000 7f1362044640 1 -- 172.21.15.73:0/2210294037 wait complete. 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 Processor -- start 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 -- start start 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100460 0x7f135c0fff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100e20 0x7f135c19c250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c10bef0 con 0x7f135c1018a0 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c10bd70 con 0x7f135c100e20 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f1362044640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f135c10c070 con 0x7f135c100460 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f135bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:58.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f135bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58342/0 (socket says 172.21.15.73:58342) 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.706+0000 7f135bfff640 1 -- 172.21.15.73:0/1516831539 learned_addr learned my addr 172.21.15.73:0/1516831539 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135b7fe640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100460 0x7f135c0fff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f1353fff640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100e20 0x7f135c19c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 -- 172.21.15.73:0/1516831539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100460 msgr2=0x7f135c0fff20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100460 0x7f135c0fff20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 -- 172.21.15.73:0/1516831539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100e20 msgr2=0x7f135c19c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100e20 0x7f135c19c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 -- 172.21.15.73:0/1516831539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f135c19f180 con 0x7f135c1018a0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f135bfff640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f135400eb10 tx=0x7f135400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354004b20 con 0x7f135c1018a0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1354004cc0 con 0x7f135c1018a0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f135c19f470 con 0x7f135c1018a0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1354010760 con 0x7f135c1018a0 2024-03-09T00:03:58.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.707+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f135c19fa10 con 0x7f135c1018a0 2024-03-09T00:03:58.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.708+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f135c107540 con 0x7f135c1018a0 2024-03-09T00:03:58.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.709+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1354005020 con 0x7f135c1018a0 2024-03-09T00:03:58.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.709+0000 7f13597fa640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 0x7f13480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:03:58.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.710+0000 7f135b7fe640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 0x7f13480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:03:58.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.710+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f13540cecc0 con 0x7f135c1018a0 2024-03-09T00:03:58.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.710+0000 7f135b7fe640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 0x7f13480a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f134c006010 tx=0x7f134c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:03:58.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.714+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f135409c200 con 0x7f135c1018a0 2024-03-09T00:03:58.762 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.762 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.762 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: Deploying daemon mds.i on smithi179 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6836/306199757,v1:172.21.15.102:6837/306199757] up:boot 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: fsmap 8 up:standby 2024-03-09T00:03:58.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:58 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: Deploying daemon mds.i on smithi179 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:58.811 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6836/306199757,v1:172.21.15.102:6837/306199757] up:boot 2024-03-09T00:03:58.812 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: fsmap 8 up:standby 2024-03-09T00:03:58.812 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:58 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:03:58.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: Deploying daemon mds.i on smithi179 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6836/306199757,v1:172.21.15.102:6837/306199757] up:boot 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: fsmap 8 up:standby 2024-03-09T00:03:58.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:58 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:03:58.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.925+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f135c061ef0 con 0x7f135c1018a0 2024-03-09T00:03:58.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.926+0000 7f13597fa640 1 -- 172.21.15.73:0/1516831539 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6874 (secure 0 0 0) 0x7f135409bb00 con 0x7f135c1018a0 2024-03-09T00:03:58.921 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:03:58.921 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-09T00:03:58.921 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 9 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.927+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 msgr2=0x7f13480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.927+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 0x7f13480a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f134c006010 tx=0x7f134c000bc0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.927+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 msgr2=0x7f135c19e8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.927+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f135400eb10 tx=0x7f135400c2d0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 shutdown_connections 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f13480a2880 0x7f13480a4d40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f135c1018a0 0x7f135c19e8c0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f135c100e20 0x7f135c19c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 --2- 172.21.15.73:0/1516831539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f135c100460 0x7f135c0fff20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 >> 172.21.15.73:0/1516831539 conn(0x7f135c0faf10 msgr2=0x7f135c104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 shutdown_connections 2024-03-09T00:03:58.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:03:58.928+0000 7f1362044640 1 -- 172.21.15.73:0/1516831539 wait complete. 2024-03-09T00:03:59.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:03:59 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1516831539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:59.817 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:03:59 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1516831539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:03:59.818 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:03:59 smithi179 systemd[1]: Starting Ceph mds.i for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:03:59.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:03:59 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1516831539' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:00.081 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:03:59 smithi179 podman[67576]: 2024-03-09 00:03:59.726264257 +0000 UTC m=+0.018152522 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:04:00.081 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:03:59 smithi179 podman[67576]: 2024-03-09 00:03:59.853293427 +0000 UTC m=+0.145181718 container create 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:04:00.502 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 podman[67576]: 2024-03-09 00:04:00.13201904 +0000 UTC m=+0.423907292 container init 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-09T00:04:00.503 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 podman[67576]: 2024-03-09 00:04:00.134870238 +0000 UTC m=+0.426758493 container start 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:04:00.503 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mds[67595]: -- 172.21.15.179:0/4115468534 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f62845a40 con 0x556f62819000 2024-03-09T00:04:00.503 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: starting mds.i at 2024-03-09T00:04:00.503 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 bash[67576]: 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde 2024-03-09T00:04:00.503 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:04:00 smithi179 systemd[1]: Started Ceph mds.i for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:00.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:00 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:00.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:00 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:00.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:00.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:00.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:00 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:00.935 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 -- 172.21.15.73:0/1268628698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101de0 msgr2=0x7f1030108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 --2- 172.21.15.73:0/1268628698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101de0 0x7f1030108670 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f1028009ca0 tx=0x7f102802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 -- 172.21.15.73:0/1268628698 shutdown_connections 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 --2- 172.21.15.73:0/1268628698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101de0 0x7f1030108670 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 --2- 172.21.15.73:0/1268628698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101420 0x7f10301018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 --2- 172.21.15.73:0/1268628698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f1030100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 -- 172.21.15.73:0/1268628698 >> 172.21.15.73:0/1268628698 conn(0x7f10300fb7a0 msgr2=0x7f10300fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 -- 172.21.15.73:0/1268628698 shutdown_connections 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.043+0000 7f10383b9640 1 -- 172.21.15.73:0/1268628698 wait complete. 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 Processor -- start 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 -- start start 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f103006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 0x7f10301a0c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010bec0 con 0x7f1030101420 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010bd40 con 0x7f1030101de0 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f10383b9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f103010c040 con 0x7f1030100230 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f1035954640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f1035954640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58348/0 (socket says 172.21.15.73:58348) 2024-03-09T00:04:01.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f1035954640 1 -- 172.21.15.73:0/1260454491 learned_addr learned my addr 172.21.15.73:0/1260454491 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.044+0000 7f1036956640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 0x7f10301a0c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1036155640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f103006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 -- 172.21.15.73:0/1260454491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 msgr2=0x7f103006a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f103006a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 -- 172.21.15.73:0/1260454491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 msgr2=0x7f10301a0c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 0x7f10301a0c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 -- 172.21.15.73:0/1260454491 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10301a1450 con 0x7f1030101420 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1036155640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f103006a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1036956640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 0x7f10301a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f1035954640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d950 tx=0x7f102c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c017070 con 0x7f1030101420 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10301a1740 con 0x7f1030101420 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f102c009c30 con 0x7f1030101420 2024-03-09T00:04:01.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c00f6b0 con 0x7f1030101420 2024-03-09T00:04:01.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.045+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10301a1ce0 con 0x7f1030101420 2024-03-09T00:04:01.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.046+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10301074f0 con 0x7f1030101420 2024-03-09T00:04:01.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.047+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f102c0109d0 con 0x7f1030101420 2024-03-09T00:04:01.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.047+0000 7f10277fe640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 0x7f101c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:01.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.047+0000 7f1036155640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 0x7f101c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:01.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.048+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f102c0ce2e0 con 0x7f1030101420 2024-03-09T00:04:01.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.048+0000 7f1036155640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 0x7f101c0a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1030101280 tx=0x7f1020005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:01.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.052+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f102c009de0 con 0x7f1030101420 2024-03-09T00:04:01.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.270+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f103002d070 con 0x7f1030101420 2024-03-09T00:04:01.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.271+0000 7f10277fe640 1 -- 172.21.15.73:0/1260454491 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7670 (secure 0 0 0) 0x7f102c09b820 con 0x7f1030101420 2024-03-09T00:04:01.267 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:01.267 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-09T00:04:01.267 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 10 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 msgr2=0x7f101c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 0x7f101c0a4cc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f1030101280 tx=0x7f1020005e70 comp rx=0 tx=0).stop 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 msgr2=0x7f103019e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f102c00d950 tx=0x7f102c00de10 comp rx=0 tx=0).stop 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 shutdown_connections 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f101c0a2800 0x7f101c0a4cc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1030101de0 0x7f10301a0c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1030101420 0x7f103019e730 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 --2- 172.21.15.73:0/1260454491 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1030100230 0x7f103006a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:01.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.273+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 >> 172.21.15.73:0/1260454491 conn(0x7f10300fb7a0 msgr2=0x7f10301046c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:01.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.274+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 shutdown_connections 2024-03-09T00:04:01.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:01.274+0000 7f10383b9640 1 -- 172.21.15.73:0/1260454491 wait complete. 2024-03-09T00:04:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:01 smithi102 ceph-mon[36106]: Deploying daemon mds.g on smithi073 2024-03-09T00:04:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:01 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.179:6838/2754171892,v1:172.21.15.179:6839/2754171892] up:boot 2024-03-09T00:04:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:01 smithi102 ceph-mon[36106]: fsmap 9 up:standby 2024-03-09T00:04:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:01 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:04:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:01 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1260454491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:01 smithi179 ceph-mon[36637]: Deploying daemon mds.g on smithi073 2024-03-09T00:04:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:01 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.179:6838/2754171892,v1:172.21.15.179:6839/2754171892] up:boot 2024-03-09T00:04:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:01 smithi179 ceph-mon[36637]: fsmap 9 up:standby 2024-03-09T00:04:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:01 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:04:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:01 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1260454491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:01.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:01 smithi073 ceph-mon[32309]: Deploying daemon mds.g on smithi073 2024-03-09T00:04:01.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:01 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.179:6838/2754171892,v1:172.21.15.179:6839/2754171892] up:boot 2024-03-09T00:04:01.901 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:01 smithi073 ceph-mon[32309]: fsmap 9 up:standby 2024-03-09T00:04:01.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:01 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:04:01.902 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:01 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1260454491' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:02.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:02 smithi102 ceph-mon[36106]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:02.814 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:02 smithi073 ceph-mon[32309]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:02.814 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:02 smithi073 systemd[1]: Starting Ceph mds.g for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:04:02.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:02 smithi179 ceph-mon[36637]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:03.096 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:02 smithi073 podman[74202]: 2024-03-09 00:04:02.722373921 +0000 UTC m=+0.017625754 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:04:03.097 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:02 smithi073 podman[74202]: 2024-03-09 00:04:02.866390968 +0000 UTC m=+0.161642871 container create aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-09T00:04:03.097 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 podman[74202]: 2024-03-09 00:04:03.102449544 +0000 UTC m=+0.397701375 container init aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:04:03.280 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:03.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 -- 172.21.15.73:0/2748950281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f841028e0 msgr2=0x7f8f84102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:03.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 --2- 172.21.15.73:0/2748950281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f841028e0 0x7f8f84102d00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8f74009900 tx=0x7f8f7402bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:03.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 -- 172.21.15.73:0/2748950281 shutdown_connections 2024-03-09T00:04:03.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 --2- 172.21.15.73:0/2748950281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f840ffd10 0x7f8f841098c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 --2- 172.21.15.73:0/2748950281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ff350 0x7f8f840ff7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 --2- 172.21.15.73:0/2748950281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f841028e0 0x7f8f84102d00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 -- 172.21.15.73:0/2748950281 >> 172.21.15.73:0/2748950281 conn(0x7f8f840faef0 msgr2=0x7f8f840fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 -- 172.21.15.73:0/2748950281 shutdown_connections 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 -- 172.21.15.73:0/2748950281 wait complete. 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.393+0000 7f8f89389640 1 Processor -- start 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 -- start start 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 0x7f8f84197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 0x7f8f8419ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8410cfe0 con 0x7f8f840ff350 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8410ce60 con 0x7f8f840ffd10 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f89389640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8410d160 con 0x7f8f841028e0 2024-03-09T00:04:03.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58384/0 (socket says 172.21.15.73:58384) 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 -- 172.21.15.73:0/574953 learned_addr learned my addr 172.21.15.73:0/574953 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f7bfff640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 0x7f8f84197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f837fe640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 0x7f8f8419ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 -- 172.21.15.73:0/574953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 msgr2=0x7f8f8419ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 0x7f8f8419ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 -- 172.21.15.73:0/574953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 msgr2=0x7f8f84197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 0x7f8f84197bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 -- 172.21.15.73:0/574953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f8419f2d0 con 0x7f8f840ff350 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f837fe640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 0x7f8f8419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f7bfff640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 0x7f8f84197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.394+0000 7f8f82ffd640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8f7402efd0 tx=0x7f8f740040c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.395+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f74041070 con 0x7f8f840ff350 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.395+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8419f560 con 0x7f8f840ff350 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.395+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8f740044c0 con 0x7f8f840ff350 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.395+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8f740346a0 con 0x7f8f840ff350 2024-03-09T00:04:03.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.395+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f8419faa0 con 0x7f8f840ff350 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.396+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f84108870 con 0x7f8f840ff350 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.397+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8f74044070 con 0x7f8f840ff350 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.397+0000 7f8f80ff9640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 0x7f8f700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.397+0000 7f8f7bfff640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 0x7f8f700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.397+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f8f740f0520 con 0x7f8f840ff350 2024-03-09T00:04:03.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.398+0000 7f8f7bfff640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 0x7f8f700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8f84198bc0 tx=0x7f8f6c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:03.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.402+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8f740bda60 con 0x7f8f840ff350 2024-03-09T00:04:03.401 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 podman[74202]: 2024-03-09 00:04:03.105539112 +0000 UTC m=+0.400790941 container start aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-09T00:04:03.401 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 ceph-mds[74221]: -- 172.21.15.73:0/979239600 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56105030fa40 con 0x5610502e2c00 2024-03-09T00:04:03.401 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g[74217]: starting mds.g at 2024-03-09T00:04:03.401 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 bash[74202]: aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 2024-03-09T00:04:03.401 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:04:03 smithi073 systemd[1]: Started Ceph mds.g for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:04:03.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.624+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f8402ce70 con 0x7f8f840ff350 2024-03-09T00:04:03.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.625+0000 7f8f80ff9640 1 -- 172.21.15.73:0/574953 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8462 (secure 0 0 0) 0x7f8f740bd360 con 0x7f8f840ff350 2024-03-09T00:04:03.620 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:03.620 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-09T00:04:03.620 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 11 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 msgr2=0x7f8f700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 0x7f8f700a4cc0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f8f84198bc0 tx=0x7f8f6c008040 comp rx=0 tx=0).stop 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 msgr2=0x7f8f841976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8f7402efd0 tx=0x7f8f740040c0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 shutdown_connections 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8f700a2800 0x7f8f700a4cc0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8f841028e0 0x7f8f8419ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8f840ffd10 0x7f8f84197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 --2- 172.21.15.73:0/574953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8f840ff350 0x7f8f841976b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:03.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 >> 172.21.15.73:0/574953 conn(0x7f8f840faef0 msgr2=0x7f8f84105a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:03.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 shutdown_connections 2024-03-09T00:04:03.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:03.627+0000 7f8f89389640 1 -- 172.21.15.73:0/574953 wait complete. 2024-03-09T00:04:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: Deploying daemon mds.a on smithi073 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.73:6838/2046390468,v1:172.21.15.73:6839/2046390468] up:boot 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: fsmap 10 up:standby 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:04:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/574953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:04 smithi102 ceph-mon[36106]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:04.581 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: Deploying daemon mds.a on smithi073 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.73:6838/2046390468,v1:172.21.15.73:6839/2046390468] up:boot 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: fsmap 10 up:standby 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/574953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:04 smithi073 ceph-mon[32309]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:04.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: Deploying daemon mds.a on smithi073 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.73:6838/2046390468,v1:172.21.15.73:6839/2046390468] up:boot 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: fsmap 10 up:standby 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/574953' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:04 smithi179 ceph-mon[36637]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:05.568 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 systemd[1]: Starting Ceph mds.a for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:04:05.633 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- 172.21.15.73:0/925159596 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c1046c0 msgr2=0x7fd89c106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/925159596 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c1046c0 0x7fd89c106ab0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fd884009900 tx=0x7fd88402bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- 172.21.15.73:0/925159596 shutdown_connections 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/925159596 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c106ff0 0x7fd89c109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/925159596 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c1046c0 0x7fd89c106ab0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/925159596 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 0x7fd89c104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- 172.21.15.73:0/925159596 >> 172.21.15.73:0/925159596 conn(0x7fd89c0fb780 msgr2=0x7fd89c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- 172.21.15.73:0/925159596 shutdown_connections 2024-03-09T00:04:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- 172.21.15.73:0/925159596 wait complete. 2024-03-09T00:04:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 Processor -- start 2024-03-09T00:04:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.746+0000 7fd8a20f8640 1 -- start start 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 0x7fd89c101700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c1046c0 0x7fd89c0ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c10cfe0 con 0x7fd89c106ff0 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c10ce60 con 0x7fd89c1046c0 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd8a20f8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89c10d160 con 0x7fd89c101d90 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58396/0 (socket says 172.21.15.73:58396) 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 -- 172.21.15.73:0/3853740126 learned_addr learned my addr 172.21.15.73:0/3853740126 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89affd640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c1046c0 0x7fd89c0ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89b7fe640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 0x7fd89c101700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 -- 172.21.15.73:0/3853740126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 msgr2=0x7fd89c101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 0x7fd89c101700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 -- 172.21.15.73:0/3853740126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c1046c0 msgr2=0x7fd89c0ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c1046c0 0x7fd89c0ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 -- 172.21.15.73:0/3853740126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd89c100b10 con 0x7fd89c106ff0 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd89bfff640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c970 tx=0x7fd88c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.747+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c007bb0 con 0x7fd89c106ff0 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.748+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd89c1a1750 con 0x7fd89c106ff0 2024-03-09T00:04:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.748+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88c007d50 con 0x7fd89c106ff0 2024-03-09T00:04:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.748+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88c018800 con 0x7fd89c106ff0 2024-03-09T00:04:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.748+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd89c1a1c20 con 0x7fd89c106ff0 2024-03-09T00:04:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.749+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd89c0680e0 con 0x7fd89c106ff0 2024-03-09T00:04:05.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.750+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fd88c00fa50 con 0x7fd89c106ff0 2024-03-09T00:04:05.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.750+0000 7fd898ff9640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 0x7fd8880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:05.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.750+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd88c0ce880 con 0x7fd89c106ff0 2024-03-09T00:04:05.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.750+0000 7fd89b7fe640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 0x7fd8880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:05.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.751+0000 7fd89b7fe640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 0x7fd8880a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd894009f20 tx=0x7fd8940094f0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:05.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.754+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd88c014030 con 0x7fd89c106ff0 2024-03-09T00:04:05.846 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 podman[74551]: 2024-03-09 00:04:05.476795898 +0000 UTC m=+0.017971376 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:04:05.846 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 podman[74551]: 2024-03-09 00:04:05.592922823 +0000 UTC m=+0.134098328 container create 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-09T00:04:05.846 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 podman[74551]: 2024-03-09 00:04:05.851669185 +0000 UTC m=+0.392844664 container init 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:04:05.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.983+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd89c0fd270 con 0x7fd89c106ff0 2024-03-09T00:04:05.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.983+0000 7fd898ff9640 1 -- 172.21.15.73:0/3853740126 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7fd89c0fd270 con 0x7fd89c106ff0 2024-03-09T00:04:05.979 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:05.979 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-09T00:04:05.979 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 12 2024-03-09T00:04:05.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 msgr2=0x7fd8880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:05.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 0x7fd8880a4cc0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fd894009f20 tx=0x7fd8940094f0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 msgr2=0x7fd89c100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd88c00c970 tx=0x7fd88c00ce30 comp rx=0 tx=0).stop 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 shutdown_connections 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fd8880a2800 0x7fd8880a4cc0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd89c106ff0 0x7fd89c100280 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd89c1046c0 0x7fd89c0ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 --2- 172.21.15.73:0/3853740126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd89c101d90 0x7fd89c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 >> 172.21.15.73:0/3853740126 conn(0x7fd89c0fb780 msgr2=0x7fd89c1026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 shutdown_connections 2024-03-09T00:04:05.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:05.987+0000 7fd8a20f8640 1 -- 172.21.15.73:0/3853740126 wait complete. 2024-03-09T00:04:06.151 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 podman[74551]: 2024-03-09 00:04:05.85518173 +0000 UTC m=+0.396357201 container start 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240307) 2024-03-09T00:04:06.151 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 ceph-mds[74609]: -- 172.21.15.73:0/2084786059 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b4241a1a40 con 0x55b424174c00 2024-03-09T00:04:06.152 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a[74586]: starting mds.a at 2024-03-09T00:04:06.152 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 bash[74551]: 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e 2024-03-09T00:04:06.152 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:04:05 smithi073 systemd[1]: Started Ceph mds.a for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:04:07.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:07.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.73:6840/2662738077,v1:172.21.15.73:6841/2662738077] up:boot 2024-03-09T00:04:07.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: fsmap 11 up:standby 2024-03-09T00:04:07.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3853740126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:07.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:06 smithi102 ceph-mon[36106]: Deploying daemon mds.b on smithi102 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.73:6840/2662738077,v1:172.21.15.73:6841/2662738077] up:boot 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: fsmap 11 up:standby 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3853740126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:07.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:06 smithi179 ceph-mon[36637]: Deploying daemon mds.b on smithi102 2024-03-09T00:04:07.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:07.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.73:6840/2662738077,v1:172.21.15.73:6841/2662738077] up:boot 2024-03-09T00:04:07.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: fsmap 11 up:standby 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3853740126' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:07.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:06 smithi073 ceph-mon[32309]: Deploying daemon mds.b on smithi102 2024-03-09T00:04:07.999 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:08.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 -- 172.21.15.73:0/1423429385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a881013f0 msgr2=0x7f1a88101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:08.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 --2- 172.21.15.73:0/1423429385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a881013f0 0x7f1a88101870 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f1a70009900 tx=0x7f1a7002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:08.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 -- 172.21.15.73:0/1423429385 shutdown_connections 2024-03-09T00:04:08.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 --2- 172.21.15.73:0/1423429385 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a88101db0 0x7f1a88108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 --2- 172.21.15.73:0/1423429385 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a881013f0 0x7f1a88101870 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 --2- 172.21.15.73:0/1423429385 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88100200 0x7f1a88100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.122 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 -- 172.21.15.73:0/1423429385 >> 172.21.15.73:0/1423429385 conn(0x7f1a880fb770 msgr2=0x7f1a880fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 -- 172.21.15.73:0/1423429385 shutdown_connections 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.120+0000 7f1a8d969640 1 -- 172.21.15.73:0/1423429385 wait complete. 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 Processor -- start 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 -- start start 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:08.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a881013f0 0x7f1a881027d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88101db0 0x7f1a88102d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8810be90 con 0x7f1a88100200 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8810bd10 con 0x7f1a881013f0 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a8d969640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a8810c010 con 0x7f1a88101db0 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a86ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a86ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:58438/0 (socket says 172.21.15.73:58438) 2024-03-09T00:04:08.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a86ffd640 1 -- 172.21.15.73:0/2784806097 learned_addr learned my addr 172.21.15.73:0/2784806097 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a7e7fc640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a881013f0 0x7f1a881027d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.121+0000 7f1a877fe640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88101db0 0x7f1a88102d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 -- 172.21.15.73:0/2784806097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88101db0 msgr2=0x7f1a88102d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88101db0 0x7f1a88102d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 -- 172.21.15.73:0/2784806097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a881013f0 msgr2=0x7f1a881027d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:08.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a881013f0 0x7f1a881027d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 -- 172.21.15.73:0/2784806097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a881035a0 con 0x7f1a88100200 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a86ffd640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800d950 tx=0x7f1a7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a78017070 con 0x7f1a88100200 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a881a1720 con 0x7f1a88100200 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a78009c30 con 0x7f1a88100200 2024-03-09T00:04:08.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a7800f600 con 0x7f1a88100200 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.122+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a881a1bf0 con 0x7f1a88100200 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.123+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a881a1eb0 con 0x7f1a88100200 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.124+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1a7801a070 con 0x7f1a88100200 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.124+0000 7f1a8c967640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 0x7f1a740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.125+0000 7f1a7e7fc640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 0x7f1a740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:08.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.125+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1a780cdf70 con 0x7f1a88100200 2024-03-09T00:04:08.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.125+0000 7f1a7e7fc640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 0x7f1a740a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a7002c7b0 tx=0x7f1a70005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:08.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.129+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1a78009de0 con 0x7f1a88100200 2024-03-09T00:04:08.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.340+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1a881a2080 con 0x7f1a88100200 2024-03-09T00:04:08.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.341+0000 7f1a8c967640 1 -- 172.21.15.73:0/2784806097 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9254 (secure 0 0 0) 0x7f1a7809ae60 con 0x7f1a88100200 2024-03-09T00:04:08.336 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:08.336 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-09T00:04:08.337 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 12 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 msgr2=0x7f1a740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 0x7f1a740a4cc0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f1a7002c7b0 tx=0x7f1a70005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 msgr2=0x7f1a88104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f1a7800d950 tx=0x7f1a7800de10 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 shutdown_connections 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1a740a2800 0x7f1a740a4cc0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1a88101db0 0x7f1a88102d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1a881013f0 0x7f1a881027d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 --2- 172.21.15.73:0/2784806097 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1a88100200 0x7f1a88104190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 >> 172.21.15.73:0/2784806097 conn(0x7f1a880fb770 msgr2=0x7f1a880fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 shutdown_connections 2024-03-09T00:04:08.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:08.343+0000 7f1a8d969640 1 -- 172.21.15.73:0/2784806097 wait complete. 2024-03-09T00:04:08.759 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 podman[66482]: 2024-03-09 00:04:08.642309982 +0000 UTC m=+0.134048486 container create bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:08 smithi102 ceph-mon[36106]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:08 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2784806097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 podman[66482]: 2024-03-09 00:04:08.887419986 +0000 UTC m=+0.379158486 container init bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 podman[66482]: 2024-03-09 00:04:08.890397155 +0000 UTC m=+0.382135634 container start bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 ceph-mds[66501]: -- 172.21.15.102:0/2867456552 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56270b1ffa40 con 0x56270b1d2800 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: starting mds.b at 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 bash[66482]: bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d 2024-03-09T00:04:09.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:08 smithi102 systemd[1]: Started Ceph mds.b for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:04:09.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:08 smithi179 ceph-mon[36637]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:09.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:08 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2784806097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:08 smithi073 ceph-mon[32309]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:09.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:08 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2784806097' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:10.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:boot 2024-03-09T00:04:10.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: fsmap 12 up:standby 2024-03-09T00:04:10.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:04:10.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:09 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:10.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:boot 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: fsmap 12 up:standby 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.275 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:09 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:10.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:boot 2024-03-09T00:04:10.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: fsmap 12 up:standby 2024-03-09T00:04:10.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:04:10.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.283 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:10.284 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:09 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:10.350 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 -- 172.21.15.73:0/450757626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 msgr2=0x7f1ae4102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 --2- 172.21.15.73:0/450757626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4102980 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009900 tx=0x7f1ae002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 -- 172.21.15.73:0/450757626 shutdown_connections 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 --2- 172.21.15.73:0/450757626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4102ec0 0x7f1ae4109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 --2- 172.21.15.73:0/450757626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4102980 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 --2- 172.21.15.73:0/450757626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4101310 0x7f1ae4101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 -- 172.21.15.73:0/450757626 >> 172.21.15.73:0/450757626 conn(0x7f1ae40fc8a0 msgr2=0x7f1ae40fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 -- 172.21.15.73:0/450757626 shutdown_connections 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.460+0000 7f1aec77e640 1 -- 172.21.15.73:0/450757626 wait complete. 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 Processor -- start 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 -- start start 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4101310 0x7f1ae41975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 0x7f1ae419eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae410cfa0 con 0x7f1ae4102500 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae410ce20 con 0x7f1ae4102ec0 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aec77e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ae410d120 con 0x7f1ae4101310 2024-03-09T00:04:10.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47388/0 (socket says 172.21.15.73:47388) 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 -- 172.21.15.73:0/913003514 learned_addr learned my addr 172.21.15.73:0/913003514 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aead1b640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 0x7f1ae419eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aea51a640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4101310 0x7f1ae41975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 -- 172.21.15.73:0/913003514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4101310 msgr2=0x7f1ae41975f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4101310 0x7f1ae41975f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 -- 172.21.15.73:0/913003514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 msgr2=0x7f1ae419eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 0x7f1ae419eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 -- 172.21.15.73:0/913003514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ae419f210 con 0x7f1ae4102500 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1aead1b640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 0x7f1ae419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.461+0000 7f1ae9d19640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009230 tx=0x7f1ae0005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:10.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.462+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0041070 con 0x7f1ae4102500 2024-03-09T00:04:10.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.462+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ae419f4a0 con 0x7f1ae4102500 2024-03-09T00:04:10.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.462+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ae00044d0 con 0x7f1ae4102500 2024-03-09T00:04:10.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.462+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ae0034440 con 0x7f1ae4102500 2024-03-09T00:04:10.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.462+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ae419f990 con 0x7f1ae4102500 2024-03-09T00:04:10.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.463+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ae4108630 con 0x7f1ae4102500 2024-03-09T00:04:10.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.464+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1ae0044070 con 0x7f1ae4102500 2024-03-09T00:04:10.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.464+0000 7f1ad77fe640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.464+0000 7f1aea51a640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.464+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1ae00f0240 con 0x7f1ae4102500 2024-03-09T00:04:10.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.465+0000 7f1aea51a640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4102360 tx=0x7f1ad8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:10.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.469+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1ae00bd780 con 0x7f1ae4102500 2024-03-09T00:04:10.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.711+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1ae402ce70 con 0x7f1ae4102500 2024-03-09T00:04:10.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.712+0000 7f1ad77fe640 1 -- 172.21.15.73:0/913003514 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10049 (secure 0 0 0) 0x7f1ae00bd080 con 0x7f1ae4102500 2024-03-09T00:04:10.707 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:10.708 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24515,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-09T00:04:10.708 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 13 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 msgr2=0x7f1ad00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f1ae4102360 tx=0x7f1ad8005f90 comp rx=0 tx=0).stop 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 msgr2=0x7f1ae4197b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f1ae0009230 tx=0x7f1ae0005d50 comp rx=0 tx=0).stop 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 shutdown_connections 2024-03-09T00:04:10.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1ad00a2800 0x7f1ad00a4cc0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1ae4102ec0 0x7f1ae419eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1ae4102500 0x7f1ae4197b30 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 --2- 172.21.15.73:0/913003514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1ae4101310 0x7f1ae41975f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 >> 172.21.15.73:0/913003514 conn(0x7f1ae40fc8a0 msgr2=0x7f1ae4105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 shutdown_connections 2024-03-09T00:04:10.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.715+0000 7f1aec77e640 1 -- 172.21.15.73:0/913003514 wait complete. 2024-03-09T00:04:10.722 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 -- 172.21.15.73:0/1301655514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738100030 msgr2=0x7fa738100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 --2- 172.21.15.73:0/1301655514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738100030 0x7fa738100450 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fa728009900 tx=0x7fa72802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 -- 172.21.15.73:0/1301655514 shutdown_connections 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 --2- 172.21.15.73:0/1301655514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738101be0 0x7fa738108470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 --2- 172.21.15.73:0/1301655514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101220 0x7fa7381016a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 --2- 172.21.15.73:0/1301655514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738100030 0x7fa738100450 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 -- 172.21.15.73:0/1301655514 >> 172.21.15.73:0/1301655514 conn(0x7fa7380fb5a0 msgr2=0x7fa7380fd9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:10.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 -- 172.21.15.73:0/1301655514 shutdown_connections 2024-03-09T00:04:10.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.840+0000 7fa73c803640 1 -- 172.21.15.73:0/1301655514 wait complete. 2024-03-09T00:04:10.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.841+0000 7fa73c803640 1 Processor -- start 2024-03-09T00:04:10.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.841+0000 7fa73c803640 1 -- start start 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.841+0000 7fa73c803640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738100030 0x7fa738196360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 0x7fa73819d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73810bc80 con 0x7fa738101220 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73810bb00 con 0x7fa738101be0 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa73810be00 con 0x7fa738100030 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47400/0 (socket says 172.21.15.73:47400) 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 -- 172.21.15.73:0/1712308626 learned_addr learned my addr 172.21.15.73:0/1712308626 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73759e640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 0x7fa73819d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa736d9d640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738100030 0x7fa738196360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 -- 172.21.15.73:0/1712308626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738100030 msgr2=0x7fa738196360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738100030 0x7fa738196360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 -- 172.21.15.73:0/1712308626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 msgr2=0x7fa73819d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 0x7fa73819d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 -- 172.21.15.73:0/1712308626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa73819df80 con 0x7fa738101220 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73759e640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 0x7fa73819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73659c640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa72c00e9f0 tx=0x7fa72c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c00cd80 con 0x7fa738101220 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa73819e270 con 0x7fa738101220 2024-03-09T00:04:10.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa72c010910 con 0x7fa738101220 2024-03-09T00:04:10.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa72c0188f0 con 0x7fa738101220 2024-03-09T00:04:10.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.842+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa73819e810 con 0x7fa738101220 2024-03-09T00:04:10.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.843+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa738107370 con 0x7fa738101220 2024-03-09T00:04:10.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.845+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa72c010ac0 con 0x7fa738101220 2024-03-09T00:04:10.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.845+0000 7fa71ffff640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 0x7fa7200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:10.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.846+0000 7fa736d9d640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 0x7fa7200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:10.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.846+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa72c0ced40 con 0x7fa738101220 2024-03-09T00:04:10.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.846+0000 7fa736d9d640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 0x7fa7200a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa728000c00 tx=0x7fa728005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:10.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:10.849+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa72c09bc30 con 0x7fa738101220 2024-03-09T00:04:11.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.042+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa73802d040 con 0x7fa738101220 2024-03-09T00:04:11.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.043+0000 7fa71ffff640 1 -- 172.21.15.73:0/1712308626 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+18443 (secure 0 0 0) 0x7fa72c09fcd0 con 0x7fa738101220 2024-03-09T00:04:11.038 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:11.038 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":81,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:03:22.460611+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":41,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":53,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":75,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:11.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 msgr2=0x7fa7200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 0x7fa7200a4cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fa728000c00 tx=0x7fa728005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 msgr2=0x7fa7381968a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7fa72c00e9f0 tx=0x7fa72c00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 shutdown_connections 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa7200a2800 0x7fa7200a4cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa738101be0 0x7fa73819d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa738101220 0x7fa7381968a0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 --2- 172.21.15.73:0/1712308626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa738100030 0x7fa738196360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 >> 172.21.15.73:0/1712308626 conn(0x7fa7380fb5a0 msgr2=0x7fa738104540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 shutdown_connections 2024-03-09T00:04:11.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.045+0000 7fa73c803640 1 -- 172.21.15.73:0/1712308626 wait complete. 2024-03-09T00:04:11.051 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-09T00:04:11.052 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:04:11.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 -- 172.21.15.73:0/109195597 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 msgr2=0x7f9128109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 --2- 172.21.15.73:0/109195597 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f9128109740 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f9120009ca0 tx=0x7f912002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 -- 172.21.15.73:0/109195597 shutdown_connections 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 --2- 172.21.15.73:0/109195597 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f9128109740 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 --2- 172.21.15.73:0/109195597 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f91281024f0 0x7f9128102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 --2- 172.21.15.73:0/109195597 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9128101300 0x7f9128101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 -- 172.21.15.73:0/109195597 >> 172.21.15.73:0/109195597 conn(0x7f91280fc890 msgr2=0x7f91280fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 -- 172.21.15.73:0/109195597 shutdown_connections 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.167+0000 7f912fa59640 1 -- 172.21.15.73:0/109195597 wait complete. 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 Processor -- start 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 -- start start 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9128101300 0x7f91281975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91281024f0 0x7f9128197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912810cf90 con 0x7f9128102eb0 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912810ce10 con 0x7f9128101300 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912fa59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f912810d110 con 0x7f91281024f0 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47416/0 (socket says 172.21.15.73:47416) 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 -- 172.21.15.73:0/1449528903 learned_addr learned my addr 172.21.15.73:0/1449528903 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912cff4640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91281024f0 0x7f9128197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912d7f5640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9128101300 0x7f91281975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 -- 172.21.15.73:0/1449528903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91281024f0 msgr2=0x7f9128197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91281024f0 0x7f9128197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 -- 172.21.15.73:0/1449528903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9128101300 msgr2=0x7f91281975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9128101300 0x7f91281975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.168+0000 7f912dff6640 1 -- 172.21.15.73:0/1449528903 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912819f1e0 con 0x7f9128102eb0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f912dff6640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9120037040 tx=0x7f9120032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120041020 con 0x7f9128102eb0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f912819f470 con 0x7f9128102eb0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9120034cd0 con 0x7f9128102eb0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9120039040 con 0x7f9128102eb0 2024-03-09T00:04:11.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.169+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f912819f960 con 0x7f9128102eb0 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.170+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9128108620 con 0x7f9128102eb0 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.171+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9120044020 con 0x7f9128102eb0 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.171+0000 7f911e7fc640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 0x7f91140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.171+0000 7f912d7f5640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 0x7f91140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.172+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f91200f0460 con 0x7f9128102eb0 2024-03-09T00:04:11.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.172+0000 7f912d7f5640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 0x7f91140a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9128102350 tx=0x7f9124005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.176+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f912004b020 con 0x7f9128102eb0 2024-03-09T00:04:11.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/913003514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:11 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1712308626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.369 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.374+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f912802d020 con 0x7f9128102eb0 2024-03-09T00:04:11.370 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.375+0000 7f911e7fc640 1 -- 172.21.15.73:0/1449528903 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v81) v1 ==== 93+0+10 (secure 0 0 0) 0x7f91200bda30 con 0x7f9128102eb0 2024-03-09T00:04:11.370 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 msgr2=0x7f91140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 0x7f91140a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f9128102350 tx=0x7f9124005f90 comp rx=0 tx=0).stop 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 msgr2=0x7f912819eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f9120037040 tx=0x7f9120032f40 comp rx=0 tx=0).stop 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 shutdown_connections 2024-03-09T00:04:11.372 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91140a2800 0x7f91140a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9128102eb0 0x7f912819eb70 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f91281024f0 0x7f9128197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 --2- 172.21.15.73:0/1449528903 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9128101300 0x7f91281975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 >> 172.21.15.73:0/1449528903 conn(0x7f91280fc890 msgr2=0x7f91281057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 shutdown_connections 2024-03-09T00:04:11.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.377+0000 7f912fa59640 1 -- 172.21.15.73:0/1449528903 wait complete. 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/913003514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.382 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.383 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.383 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:11.383 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:11.383 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.383 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:11 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1712308626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.384 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- 172.21.15.73:0/561766257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed80752b0 msgr2=0x7feed8073700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- 172.21.15.73:0/561766257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed80752b0 0x7feed8073700 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7feec8009900 tx=0x7feec802bcd0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- 172.21.15.73:0/561766257 shutdown_connections 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- 172.21.15.73:0/561766257 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed81091e0 0x7feed810b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- 172.21.15.73:0/561766257 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed8073cd0 0x7feed8074130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- 172.21.15.73:0/561766257 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed80752b0 0x7feed8073700 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- 172.21.15.73:0/561766257 >> 172.21.15.73:0/561766257 conn(0x7feed80fc8a0 msgr2=0x7feed80fece0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- 172.21.15.73:0/561766257 shutdown_connections 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- 172.21.15.73:0/561766257 wait complete. 2024-03-09T00:04:11.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 Processor -- start 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- start start 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8073cd0 0x7feed8199840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed80752b0 0x7feed8199d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed810f1d0 con 0x7feed81091e0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed810f050 con 0x7feed80752b0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.493+0000 7feedf82b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feed810f350 con 0x7feed8073cd0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47422/0 (socket says 172.21.15.73:47422) 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedd5c7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8073cd0 0x7feed8199840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 -- 172.21.15.73:0/2855177403 learned_addr learned my addr 172.21.15.73:0/2855177403 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedcdc6640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed80752b0 0x7feed8199d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 -- 172.21.15.73:0/2855177403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8073cd0 msgr2=0x7feed8199840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8073cd0 0x7feed8199840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 -- 172.21.15.73:0/2855177403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed80752b0 msgr2=0x7feed8199d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed80752b0 0x7feed8199d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 -- 172.21.15.73:0/2855177403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feed81a1460 con 0x7feed81091e0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedddc8640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feed000eb10 tx=0x7feed000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0004b20 con 0x7feed81091e0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feed0004cc0 con 0x7feed81091e0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feed81a1750 con 0x7feed81091e0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feed0010760 con 0x7feed81091e0 2024-03-09T00:04:11.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.494+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feed81a1ca0 con 0x7feed81091e0 2024-03-09T00:04:11.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.495+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feed8107fa0 con 0x7feed81091e0 2024-03-09T00:04:11.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.496+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feed0005000 con 0x7feed81091e0 2024-03-09T00:04:11.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.497+0000 7feece7fc640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 0x7feec40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.497+0000 7feedd5c7640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 0x7feec40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.497+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7feed00cf030 con 0x7feed81091e0 2024-03-09T00:04:11.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.497+0000 7feedd5c7640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 0x7feec40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feec802c850 tx=0x7feec8000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.505+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feed009c550 con 0x7feed81091e0 2024-03-09T00:04:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/913003514' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:11.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:11 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1712308626' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.697+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7feed8061ef0 con 0x7feed81091e0 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.698+0000 7feece7fc640 1 -- 172.21.15.73:0/2855177403 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v81) v1 ==== 84+0+65 (secure 0 0 0) 0x7feed009be50 con 0x7feed81091e0 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout:[ 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout: "poolnum": 1, 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout: "poolname": ".mgr" 2024-03-09T00:04:11.693 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:04:11.694 INFO:teuthology.orchestra.run.smithi073.stdout:] 2024-03-09T00:04:11.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 msgr2=0x7feec40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 0x7feec40a4cc0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feec802c850 tx=0x7feec8000bc0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 msgr2=0x7feed81a0df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7feed000eb10 tx=0x7feed000c2d0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 shutdown_connections 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7feec40a2800 0x7feec40a4cc0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feed81091e0 0x7feed81a0df0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feed80752b0 0x7feed8199d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 --2- 172.21.15.73:0/2855177403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feed8073cd0 0x7feed8199840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 >> 172.21.15.73:0/2855177403 conn(0x7feed80fc8a0 msgr2=0x7feed8105170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 shutdown_connections 2024-03-09T00:04:11.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.700+0000 7feedf82b640 1 -- 172.21.15.73:0/2855177403 wait complete. 2024-03-09T00:04:11.707 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-09T00:04:11.707 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 -- 172.21.15.73:0/3036967404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c100200 msgr2=0x7f214c100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 --2- 172.21.15.73:0/3036967404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c100200 0x7f214c100620 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f213c009900 tx=0x7f213c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 -- 172.21.15.73:0/3036967404 shutdown_connections 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 --2- 172.21.15.73:0/3036967404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c101db0 0x7f214c108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 --2- 172.21.15.73:0/3036967404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 --2- 172.21.15.73:0/3036967404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c100200 0x7f214c100620 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 -- 172.21.15.73:0/3036967404 >> 172.21.15.73:0/3036967404 conn(0x7f214c0fb790 msgr2=0x7f214c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 -- 172.21.15.73:0/3036967404 shutdown_connections 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.822+0000 7f215092b640 1 -- 172.21.15.73:0/3036967404 wait complete. 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 Processor -- start 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 -- start start 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 0x7f214c06a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c19c3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c10bed0 con 0x7f214c101db0 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c10bd50 con 0x7f214c100200 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f215092b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f214c10c050 con 0x7f214c1013f0 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47452/0 (socket says 172.21.15.73:47452) 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 -- 172.21.15.73:0/1134768322 learned_addr learned my addr 172.21.15.73:0/1134768322 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f2141d9b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c19c3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214a59c640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 0x7f214c06a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 -- 172.21.15.73:0/1134768322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 msgr2=0x7f214c19c3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c19c3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 -- 172.21.15.73:0/1134768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 msgr2=0x7f214c06a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 0x7f214c06a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214ad9d640 1 -- 172.21.15.73:0/1134768322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f214c19f2d0 con 0x7f214c101db0 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f2141d9b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.823+0000 7f214a59c640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 0x7f214c06a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f214ad9d640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f214400eaa0 tx=0x7f214400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f214400cdb0 con 0x7f214c101db0 2024-03-09T00:04:11.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f214c19f5c0 con 0x7f214c101db0 2024-03-09T00:04:11.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2144005020 con 0x7f214c101db0 2024-03-09T00:04:11.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21440116b0 con 0x7f214c101db0 2024-03-09T00:04:11.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.824+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f214c19fb10 con 0x7f214c101db0 2024-03-09T00:04:11.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.825+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f214c107640 con 0x7f214c101db0 2024-03-09T00:04:11.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.826+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2144005560 con 0x7f214c101db0 2024-03-09T00:04:11.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.826+0000 7f2143fff640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 0x7f21380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:11.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.826+0000 7f214a59c640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 0x7f21380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:11.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.826+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f21440ce8d0 con 0x7f214c101db0 2024-03-09T00:04:11.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.827+0000 7f214a59c640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 0x7f21380a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:11.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:11.831+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2144016030 con 0x7f214c101db0 2024-03-09T00:04:12.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.011+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7f214c02d070 con 0x7f214c101db0 2024-03-09T00:04:12.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.309+0000 7f2143fff640 1 -- 172.21.15.73:0/1134768322 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v82) v1 ==== 138+0+0 (secure 0 0 0) 0x7f214409be10 con 0x7f214c101db0 2024-03-09T00:04:12.304 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_metadata' created 2024-03-09T00:04:12.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 msgr2=0x7f21380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:12.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 0x7f21380a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f213c006010 tx=0x7f213c000bc0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 msgr2=0x7f214c19ea10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f214400eaa0 tx=0x7f214400ef60 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 shutdown_connections 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f21380a2800 0x7f21380a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f214c101db0 0x7f214c19ea10 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f214c1013f0 0x7f214c19c3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 --2- 172.21.15.73:0/1134768322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f214c100200 0x7f214c06a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.311+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 >> 172.21.15.73:0/1134768322 conn(0x7f214c0fb790 msgr2=0x7f214c104810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.312+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 shutdown_connections 2024-03-09T00:04:12.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.312+0000 7f215092b640 1 -- 172.21.15.73:0/1134768322 wait complete. 2024-03-09T00:04:12.320 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.05 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 -- 172.21.15.73:0/1010820230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102ff0 msgr2=0x7f9064109880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 --2- 172.21.15.73:0/1010820230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102ff0 0x7f9064109880 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f905c009cd0 tx=0x7f905c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 -- 172.21.15.73:0/1010820230 shutdown_connections 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 --2- 172.21.15.73:0/1010820230 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102ff0 0x7f9064109880 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 --2- 172.21.15.73:0/1010820230 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9064102630 0x7f9064102ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 --2- 172.21.15.73:0/1010820230 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f90640ff5c0 0x7f90640ff9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 -- 172.21.15.73:0/1010820230 >> 172.21.15.73:0/1010820230 conn(0x7f90640faed0 msgr2=0x7f90640fd310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:12.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 -- 172.21.15.73:0/1010820230 shutdown_connections 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.439+0000 7f9068b60640 1 -- 172.21.15.73:0/1010820230 wait complete. 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 Processor -- start 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 -- start start 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 0x7f9064197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9064102ff0 0x7f906419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906410cf80 con 0x7f9064102630 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906410ce00 con 0x7f90640ff5c0 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9068b60640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f906410d100 con 0x7f9064102ff0 2024-03-09T00:04:12.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47470/0 (socket says 172.21.15.73:47470) 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 -- 172.21.15.73:0/507510006 learned_addr learned my addr 172.21.15.73:0/507510006 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f906259c640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 0x7f9064197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9062d9d640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9064102ff0 0x7f906419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 -- 172.21.15.73:0/507510006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9064102ff0 msgr2=0x7f906419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9064102ff0 0x7f906419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 -- 172.21.15.73:0/507510006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 msgr2=0x7f9064197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 0x7f9064197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f9061d9b640 1 -- 172.21.15.73:0/507510006 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f906419f230 con 0x7f9064102630 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.440+0000 7f906259c640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 0x7f9064197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:04:12.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f9061d9b640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9f0 tx=0x7f905800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:12.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f905800cd80 con 0x7f9064102630 2024-03-09T00:04:12.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f906419f520 con 0x7f9064102630 2024-03-09T00:04:12.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9058010910 con 0x7f9064102630 2024-03-09T00:04:12.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f90580188f0 con 0x7f9064102630 2024-03-09T00:04:12.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.441+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f906419fa70 con 0x7f9064102630 2024-03-09T00:04:12.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.442+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9064108760 con 0x7f9064102630 2024-03-09T00:04:12.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.443+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9058010ac0 con 0x7f9064102630 2024-03-09T00:04:12.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.444+0000 7f90537fe640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 0x7f904c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:12.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.444+0000 7f906259c640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 0x7f904c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:12.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.444+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7497+0+0 (secure 0 0 0) 0x7f90580ceae0 con 0x7f9064102630 2024-03-09T00:04:12.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.444+0000 7f906259c640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 0x7f904c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9064068190 tx=0x7f9054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:12.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.449+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f905809b860 con 0x7f9064102630 2024-03-09T00:04:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:12 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:12 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1449528903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:12 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2855177403' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-09T00:04:12.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:12 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:12.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:12 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1449528903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:12.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:12 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2855177403' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-09T00:04:12.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:12 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:04:12.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:12 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1449528903' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:12.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:12 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2855177403' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-09T00:04:12.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:12.656+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05} v 0) v1 -- 0x7f906402ce70 con 0x7f9064102630 2024-03-09T00:04:13.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.313+0000 7f90537fe640 1 -- 172.21.15.73:0/507510006 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]=0 pool 'cephfs_data' created v83) v1 ==== 171+0+0 (secure 0 0 0) 0x7f905809f900 con 0x7f9064102630 2024-03-09T00:04:13.309 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_data' created 2024-03-09T00:04:13.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 msgr2=0x7f904c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 0x7f904c0a4cc0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f9064068190 tx=0x7f9054005f90 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 msgr2=0x7f9064197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f905800e9f0 tx=0x7f905800eeb0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 shutdown_connections 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f904c0a2800 0x7f904c0a4cc0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9064102ff0 0x7f906419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9064102630 0x7f9064197b50 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 --2- 172.21.15.73:0/507510006 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f90640ff5c0 0x7f9064197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 >> 172.21.15.73:0/507510006 conn(0x7f90640faed0 msgr2=0x7f9064105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 shutdown_connections 2024-03-09T00:04:13.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.315+0000 7f9068b60640 1 -- 172.21.15.73:0/507510006 wait complete. 2024-03-09T00:04:13.331 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-09T00:04:13.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 -- 172.21.15.73:0/225695560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641013e0 msgr2=0x7f7a64101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/225695560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641013e0 0x7f7a64101860 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7a54009ca0 tx=0x7f7a5402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 -- 172.21.15.73:0/225695560 shutdown_connections 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/225695560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64101da0 0x7f7a64108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/225695560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641013e0 0x7f7a64101860 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/225695560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641001f0 0x7f7a64100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 -- 172.21.15.73:0/225695560 >> 172.21.15.73:0/225695560 conn(0x7f7a640fb780 msgr2=0x7f7a640fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 -- 172.21.15.73:0/225695560 shutdown_connections 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.441+0000 7f7a6ad16640 1 -- 172.21.15.73:0/225695560 wait complete. 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 Processor -- start 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 -- start start 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a641001f0 0x7f7a641042d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:13.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 0x7f7a64102910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6410be80 con 0x7f7a64101da0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6410bd00 con 0x7f7a641013e0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a6ad16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a6410c000 con 0x7f7a641001f0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a692b3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a692b3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47494/0 (socket says 172.21.15.73:47494) 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a692b3640 1 -- 172.21.15.73:0/3909661084 learned_addr learned my addr 172.21.15.73:0/3909661084 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a68ab2640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a641001f0 0x7f7a641042d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.442+0000 7f7a63fff640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 0x7f7a64102910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 -- 172.21.15.73:0/3909661084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a641001f0 msgr2=0x7f7a641042d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a641001f0 0x7f7a641042d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 -- 172.21.15.73:0/3909661084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 msgr2=0x7f7a64102910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:13.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 0x7f7a64102910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 -- 172.21.15.73:0/3909661084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a641036e0 con 0x7f7a64101da0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a63fff640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 0x7f7a64102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a692b3640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00d750 tx=0x7f7a5c00dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c017070 con 0x7f7a64101da0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c005500 con 0x7f7a64101da0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c004d90 con 0x7f7a64101da0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a641a1740 con 0x7f7a64101da0 2024-03-09T00:04:13.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.443+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641a1c10 con 0x7f7a64101da0 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.444+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a641a1ed0 con 0x7f7a64101da0 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.445+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7a5c004f30 con 0x7f7a64101da0 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.446+0000 7f7a61ffb640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 0x7f7a500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.446+0000 7f7a68ab2640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 0x7f7a500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.446+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f7a5c0ce630 con 0x7f7a64101da0 2024-03-09T00:04:13.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.446+0000 7f7a68ab2640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 0x7f7a500a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7a64101240 tx=0x7f7a4c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:13.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.449+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7a5c09b880 con 0x7f7a64101da0 2024-03-09T00:04:13.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: osdmap e82: 12 total, 12 up, 12 in 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:13.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:13.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-09T00:04:13.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: osdmap e82: 12 total, 12 up, 12 in 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:13.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:13.659+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f7a641a22d0 con 0x7f7a64101da0 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1134768322' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: osdmap e82: 12 total, 12 up, 12 in 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]: dispatch 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:13.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:04:13.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:04:13.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:14.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.318+0000 7f7a61ffb640 1 -- 172.21.15.73:0/3909661084 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f7a5c09b180 con 0x7f7a64101da0 2024-03-09T00:04:14.313 INFO:teuthology.orchestra.run.smithi073.stderr:new fs with metadata pool 2 and data pool 3 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 msgr2=0x7f7a500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 0x7f7a500a4cc0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f7a64101240 tx=0x7f7a4c005f90 comp rx=0 tx=0).stop 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 msgr2=0x7f7a64102e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c00d750 tx=0x7f7a5c00dc10 comp rx=0 tx=0).stop 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 shutdown_connections 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7a500a2800 0x7f7a500a4cc0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a64101da0 0x7f7a64102e50 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641013e0 0x7f7a64102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 --2- 172.21.15.73:0/3909661084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a641001f0 0x7f7a641042d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 >> 172.21.15.73:0/3909661084 conn(0x7f7a640fb780 msgr2=0x7f7a64068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:14.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 shutdown_connections 2024-03-09T00:04:14.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.321+0000 7f7a6ad16640 1 -- 172.21.15.73:0/3909661084 wait complete. 2024-03-09T00:04:14.327 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: osdmap e83: 12 total, 12 up, 12 in 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: pgmap v288: 97 pgs: 14 creating+peering, 1 active+clean, 82 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:14.328 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:14 smithi073 ceph-mon[32309]: osdmap e84: 12 total, 12 up, 12 in 2024-03-09T00:04:14.329 DEBUG:tasks.cephfs.filesystem:EC profile is ['m=2', 'k=2', 'crush-failure-domain=osd'] 2024-03-09T00:04:14.329 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd erasure-code-profile set cephfs_data_ec m=2 k=2 crush-failure-domain=osd 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- 172.21.15.73:0/3053751320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac0ff5c0 msgr2=0x7efcac0ff9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/3053751320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac0ff5c0 0x7efcac0ff9e0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009930 tx=0x7efc9c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- 172.21.15.73:0/3053751320 shutdown_connections 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/3053751320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac102f50 0x7efcac1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/3053751320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 0x7efcac102a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/3053751320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac0ff5c0 0x7efcac0ff9e0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- 172.21.15.73:0/3053751320 >> 172.21.15.73:0/3053751320 conn(0x7efcac0faf10 msgr2=0x7efcac0fd330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:14.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- 172.21.15.73:0/3053751320 shutdown_connections 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- 172.21.15.73:0/3053751320 wait complete. 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 Processor -- start 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.442+0000 7efcb2ef8640 1 -- start start 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac0ff5c0 0x7efcac197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 0x7efcac197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac10d000 con 0x7efcac102f50 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac10ce80 con 0x7efcac102590 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcac10d180 con 0x7efcac0ff5c0 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47520/0 (socket says 172.21.15.73:47520) 2024-03-09T00:04:14.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 -- 172.21.15.73:0/1065269567 learned_addr learned my addr 172.21.15.73:0/1065269567 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcabfff640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 0x7efcac197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb0c94640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac0ff5c0 0x7efcac197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 -- 172.21.15.73:0/1065269567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac0ff5c0 msgr2=0x7efcac197630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac0ff5c0 0x7efcac197630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 -- 172.21.15.73:0/1065269567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 msgr2=0x7efcac197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 0x7efcac197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 -- 172.21.15.73:0/1065269567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcac19f250 con 0x7efcac102f50 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb1495640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efca400eaa0 tx=0x7efca400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca400cdb0 con 0x7efcac102f50 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca4005020 con 0x7efcac102f50 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcac19f540 con 0x7efcac102f50 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.443+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca40116b0 con 0x7efcac102f50 2024-03-09T00:04:14.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.444+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcac19fa90 con 0x7efcac102f50 2024-03-09T00:04:14.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.445+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcac108760 con 0x7efcac102f50 2024-03-09T00:04:14.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.445+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7efca4005560 con 0x7efcac102f50 2024-03-09T00:04:14.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.446+0000 7efca9ffb640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 0x7efc940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:14.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.446+0000 7efcb0c94640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 0x7efc940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:14.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.446+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efca40ceab0 con 0x7efcac102f50 2024-03-09T00:04:14.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.447+0000 7efcb0c94640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 0x7efc940a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009230 tx=0x7efc9c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:14.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.451+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efca4016030 con 0x7efcac102f50 2024-03-09T00:04:14.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: osdmap e83: 12 total, 12 up, 12 in 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: pgmap v288: 97 pgs: 14 creating+peering, 1 active+clean, 82 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:14 smithi179 ceph-mon[36637]: osdmap e84: 12 total, 12 up, 12 in 2024-03-09T00:04:14.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.634+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]} v 0) v1 -- 0x7efcac061ef0 con 0x7efcac102f50 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/507510006' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.05}]': finished 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: osdmap e83: 12 total, 12 up, 12 in 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: pgmap v288: 97 pgs: 14 creating+peering, 1 active+clean, 82 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:04:14.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:14 smithi102 ceph-mon[36106]: osdmap e84: 12 total, 12 up, 12 in 2024-03-09T00:04:14.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.913+0000 7efca9ffb640 1 -- 172.21.15.73:0/1065269567 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]=0 v85) v1 ==== 157+0+0 (secure 0 0 0) 0x7efca409b650 con 0x7efcac102f50 2024-03-09T00:04:14.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 msgr2=0x7efc940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 0x7efc940a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7efc9c009230 tx=0x7efc9c02df90 comp rx=0 tx=0).stop 2024-03-09T00:04:14.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 msgr2=0x7efcac19ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:14.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7efca400eaa0 tx=0x7efca400ef60 comp rx=0 tx=0).stop 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 shutdown_connections 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc940a2800 0x7efc940a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efcac102f50 0x7efcac19ebe0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efcac102590 0x7efcac197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 --2- 172.21.15.73:0/1065269567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efcac0ff5c0 0x7efcac197630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 >> 172.21.15.73:0/1065269567 conn(0x7efcac0faf10 msgr2=0x7efcac105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 shutdown_connections 2024-03-09T00:04:14.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:14.916+0000 7efcb2ef8640 1 -- 172.21.15.73:0/1065269567 wait complete. 2024-03-09T00:04:14.923 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data_ec erasure cephfs_data_ec --pg_num_min 64 --target_size_ratio 0.9 2024-03-09T00:04:15.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 -- 172.21.15.73:0/1052454120 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958101dc0 msgr2=0x7ff958108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 --2- 172.21.15.73:0/1052454120 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958101dc0 0x7ff958108650 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7ff950009ca0 tx=0x7ff95002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 -- 172.21.15.73:0/1052454120 shutdown_connections 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 --2- 172.21.15.73:0/1052454120 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958101dc0 0x7ff958108650 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 --2- 172.21.15.73:0/1052454120 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101400 0x7ff958101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 --2- 172.21.15.73:0/1052454120 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958100210 0x7ff958100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 -- 172.21.15.73:0/1052454120 >> 172.21.15.73:0/1052454120 conn(0x7ff9580fb780 msgr2=0x7ff9580fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.037+0000 7ff96053b640 1 -- 172.21.15.73:0/1052454120 shutdown_connections 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 -- 172.21.15.73:0/1052454120 wait complete. 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 Processor -- start 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 -- start start 2024-03-09T00:04:15.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958101400 0x7ff958197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 0x7ff95819eb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95810bea0 con 0x7ff958100210 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95810bd20 con 0x7ff958101400 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff96053b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff95810c020 con 0x7ff958101dc0 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff95e2d7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff95e2d7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47544/0 (socket says 172.21.15.73:47544) 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff95e2d7640 1 -- 172.21.15.73:0/319659187 learned_addr learned my addr 172.21.15.73:0/319659187 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.038+0000 7ff95dad6640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958101400 0x7ff958197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95ead8640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 0x7ff95819eb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 -- 172.21.15.73:0/319659187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 msgr2=0x7ff95819eb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 0x7ff95819eb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 -- 172.21.15.73:0/319659187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958101400 msgr2=0x7ff958197aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958101400 0x7ff958197aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 -- 172.21.15.73:0/319659187 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff95819f180 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95ead8640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 0x7ff95819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff95e2d7640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c970 tx=0x7ff95400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff954004220 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9540043c0 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95400f450 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff95819f470 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.039+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff95819f9c0 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.040+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff95819fc80 con 0x7ff958100210 2024-03-09T00:04:15.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.041+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff9540079b0 con 0x7ff958100210 2024-03-09T00:04:15.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.042+0000 7ff94f7fe640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 0x7ff9440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:15.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.042+0000 7ff95dad6640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 0x7ff9440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:15.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.042+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff9540ce7f0 con 0x7ff958100210 2024-03-09T00:04:15.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.042+0000 7ff95dad6640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 0x7ff9440a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff958198a70 tx=0x7ff94800a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:15.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.046+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff954014030 con 0x7ff958100210 2024-03-09T00:04:15.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:15.224+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff95819ff00 con 0x7ff958100210 2024-03-09T00:04:15.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:active 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: osdmap e85: 12 total, 12 up, 12 in 2024-03-09T00:04:15.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-09T00:04:15.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:active 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: osdmap e85: 12 total, 12 up, 12 in 2024-03-09T00:04:15.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-09T00:04:15.660 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:15.660 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3909661084' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-09T00:04:15.660 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-09T00:04:15.660 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd=[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]: dispatch 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] up:active 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1065269567' entity='client.admin' cmd='[{"prefix": "osd erasure-code-profile set", "name": "cephfs_data_ec", "profile": ["m=2", "k=2", "crush-failure-domain=osd"]}]': finished 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: osdmap e85: 12 total, 12 up, 12 in 2024-03-09T00:04:15.661 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-09T00:04:16.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.925+0000 7ff94f7fe640 1 -- 172.21.15.73:0/319659187 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data_ec' created v87) v1 ==== 228+0+0 (secure 0 0 0) 0x7ff95409b860 con 0x7ff958100210 2024-03-09T00:04:16.920 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_data_ec' created 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 msgr2=0x7ff9440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 0x7ff9440a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7ff958198a70 tx=0x7ff94800a2d0 comp rx=0 tx=0).stop 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 msgr2=0x7ff958197560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff95400c970 tx=0x7ff95400cd70 comp rx=0 tx=0).stop 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 shutdown_connections 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff9440a2800 0x7ff9440a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff958101dc0 0x7ff95819eb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff958101400 0x7ff958197aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:16.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 --2- 172.21.15.73:0/319659187 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff958100210 0x7ff958197560 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:16.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 >> 172.21.15.73:0/319659187 conn(0x7ff9580fb780 msgr2=0x7ff9580fd170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:16.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 shutdown_connections 2024-03-09T00:04:16.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:16.927+0000 7ff96053b640 1 -- 172.21.15.73:0/319659187 wait complete. 2024-03-09T00:04:16.939 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool set cephfs_data_ec allow_ec_overwrites true 2024-03-09T00:04:17.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:16 smithi102 ceph-mon[36106]: pgmap v291: 97 pgs: 14 creating+peering, 45 active+clean, 38 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-09T00:04:17.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:16 smithi102 ceph-mon[36106]: osdmap e86: 12 total, 12 up, 12 in 2024-03-09T00:04:17.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:16 smithi102 ceph-mon[36106]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-09T00:04:17.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:16 smithi102 ceph-mon[36106]: Cluster is now healthy 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1575347869 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 msgr2=0x7f3c70109320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1575347869 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c70109320 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f3c68009900 tx=0x7f3c6802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1575347869 shutdown_connections 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1575347869 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c70109320 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1575347869 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c701045a0 0x7f3c70106990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1575347869 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c70101c70 0x7f3c70104060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1575347869 >> 172.21.15.73:0/1575347869 conn(0x7f3c700fb660 msgr2=0x7f3c700fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1575347869 shutdown_connections 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1575347869 wait complete. 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 Processor -- start 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.050+0000 7f3c6ffff640 1 -- start start 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c70101c70 0x7f3c70197500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c701045a0 0x7f3c70197a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7010cee0 con 0x7f3c70106ed0 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7010cd60 con 0x7f3c701045a0 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7010d060 con 0x7f3c70101c70 2024-03-09T00:04:17.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47568/0 (socket says 172.21.15.73:47568) 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 -- 172.21.15.73:0/1746646070 learned_addr learned my addr 172.21.15.73:0/1746646070 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6e7fc640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c701045a0 0x7f3c70197a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6effd640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c70101c70 0x7f3c70197500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 -- 172.21.15.73:0/1746646070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c70101c70 msgr2=0x7f3c70197500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c70101c70 0x7f3c70197500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 -- 172.21.15.73:0/1746646070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c701045a0 msgr2=0x7f3c70197a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c701045a0 0x7f3c70197a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 -- 172.21.15.73:0/1746646070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7019f120 con 0x7f3c70106ed0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6f7fe640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3c68005ab0 tx=0x7f3c68004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c68041070 con 0x7f3c70106ed0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7019f3b0 con 0x7f3c70106ed0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c680054e0 con 0x7f3c70106ed0 2024-03-09T00:04:17.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c680346a0 con 0x7f3c70106ed0 2024-03-09T00:04:17.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.051+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7019f8f0 con 0x7f3c70106ed0 2024-03-09T00:04:17.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.052+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c700fd150 con 0x7f3c70106ed0 2024-03-09T00:04:17.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.054+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3c68044070 con 0x7f3c70106ed0 2024-03-09T00:04:17.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.054+0000 7f3c57fff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 0x7f3c580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:17.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.054+0000 7f3c6effd640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 0x7f3c580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:17.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.054+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f3c680f0fd0 con 0x7f3c70106ed0 2024-03-09T00:04:17.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.054+0000 7f3c6effd640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 0x7f3c580a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c60005c40 tx=0x7f3c6000a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:17.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.058+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3c680bde50 con 0x7f3c70106ed0 2024-03-09T00:04:17.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:16 smithi179 ceph-mon[36637]: pgmap v291: 97 pgs: 14 creating+peering, 45 active+clean, 38 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-09T00:04:17.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:16 smithi179 ceph-mon[36637]: osdmap e86: 12 total, 12 up, 12 in 2024-03-09T00:04:17.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:16 smithi179 ceph-mon[36637]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-09T00:04:17.082 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:16 smithi179 ceph-mon[36637]: Cluster is now healthy 2024-03-09T00:04:17.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.232+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"} v 0) v1 -- 0x7f3c7002ce70 con 0x7f3c70106ed0 2024-03-09T00:04:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:16 smithi073 ceph-mon[32309]: pgmap v291: 97 pgs: 14 creating+peering, 45 active+clean, 38 unknown; 577 KiB data, 369 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s wr, 2 op/s 2024-03-09T00:04:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:16 smithi073 ceph-mon[32309]: osdmap e86: 12 total, 12 up, 12 in 2024-03-09T00:04:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:16 smithi073 ceph-mon[32309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-09T00:04:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:16 smithi073 ceph-mon[32309]: Cluster is now healthy 2024-03-09T00:04:17.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.933+0000 7f3c57fff640 1 -- 172.21.15.73:0/1746646070 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]=0 set pool 4 allow_ec_overwrites to true v88) v1 ==== 169+0+0 (secure 0 0 0) 0x7f3c68003e20 con 0x7f3c70106ed0 2024-03-09T00:04:17.928 INFO:teuthology.orchestra.run.smithi073.stderr:set pool 4 allow_ec_overwrites to true 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 msgr2=0x7f3c580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 0x7f3c580a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3c60005c40 tx=0x7f3c6000a590 comp rx=0 tx=0).stop 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 msgr2=0x7f3c7019eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f3c68005ab0 tx=0x7f3c68004270 comp rx=0 tx=0).stop 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 shutdown_connections 2024-03-09T00:04:17.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3c580a2800 0x7f3c580a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3c70106ed0 0x7f3c7019eab0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3c701045a0 0x7f3c70197a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 --2- 172.21.15.73:0/1746646070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3c70101c70 0x7f3c70197500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 >> 172.21.15.73:0/1746646070 conn(0x7f3c700fb660 msgr2=0x7f3c701025d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 shutdown_connections 2024-03-09T00:04:17.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:17.935+0000 7f3c6ffff640 1 -- 172.21.15.73:0/1746646070 wait complete. 2024-03-09T00:04:17.959 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs add_data_pool cephfs cephfs_data_ec 2024-03-09T00:04:18.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- 172.21.15.73:0/3667939536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f4101da0 msgr2=0x7f15f4108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:18.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- 172.21.15.73:0/3667939536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f4101da0 0x7f15f4108630 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f15ec009ca0 tx=0x7f15ec02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- 172.21.15.73:0/3667939536 shutdown_connections 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- 172.21.15.73:0/3667939536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f4101da0 0x7f15f4108630 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- 172.21.15.73:0/3667939536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f41013e0 0x7f15f4101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- 172.21.15.73:0/3667939536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41001f0 0x7f15f4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- 172.21.15.73:0/3667939536 >> 172.21.15.73:0/3667939536 conn(0x7f15f40fb7a0 msgr2=0x7f15f40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- 172.21.15.73:0/3667939536 shutdown_connections 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- 172.21.15.73:0/3667939536 wait complete. 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 Processor -- start 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- start start 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 0x7f15f41969e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 0x7f15f419da50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f410be50 con 0x7f15f41001f0 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f410bcd0 con 0x7f15f41013e0 2024-03-09T00:04:18.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.074+0000 7f15fbaea640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15f410bfd0 con 0x7f15f4101da0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47588/0 (socket says 172.21.15.73:47588) 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 -- 172.21.15.73:0/1769347879 learned_addr learned my addr 172.21.15.73:0/1769347879 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15fa087640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 0x7f15f419da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9085640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 0x7f15f41969e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 -- 172.21.15.73:0/1769347879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 msgr2=0x7f15f419da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 0x7f15f419da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 -- 172.21.15.73:0/1769347879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 msgr2=0x7f15f41969e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 0x7f15f41969e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 -- 172.21.15.73:0/1769347879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15f419e0c0 con 0x7f15f41001f0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15fa087640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 0x7f15f419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9085640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 0x7f15f41969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15f9886640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15f000d770 tx=0x7f15f000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:18.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0017070 con 0x7f15f41001f0 2024-03-09T00:04:18.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15f0009c30 con 0x7f15f41001f0 2024-03-09T00:04:18.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15f419e3b0 con 0x7f15f41001f0 2024-03-09T00:04:18.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15f0003dd0 con 0x7f15f41001f0 2024-03-09T00:04:18.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.075+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15f419e900 con 0x7f15f41001f0 2024-03-09T00:04:18.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.076+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15f4107510 con 0x7f15f41001f0 2024-03-09T00:04:18.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.077+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f15f0003fb0 con 0x7f15f41001f0 2024-03-09T00:04:18.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.078+0000 7f15eaffd640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 0x7f15e00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:18.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.078+0000 7f15f9085640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 0x7f15e00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:18.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.078+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8770+0+0 (secure 0 0 0) 0x7f15f00cead0 con 0x7f15f41001f0 2024-03-09T00:04:18.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.078+0000 7f15f9085640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 0x7f15e00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f15f41979b0 tx=0x7f15e4005e50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:18.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.081+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f15f0009de0 con 0x7f15f41001f0 2024-03-09T00:04:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:17 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-09T00:04:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:17 smithi102 ceph-mon[36106]: osdmap e87: 12 total, 12 up, 12 in 2024-03-09T00:04:18.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:17 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-09T00:04:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.297+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"} v 0) v1 -- 0x7f15f402ce70 con 0x7f15f41001f0 2024-03-09T00:04:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:17 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-09T00:04:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:17 smithi179 ceph-mon[36637]: osdmap e87: 12 total, 12 up, 12 in 2024-03-09T00:04:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:17 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-09T00:04:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:17 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/319659187' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data_ec", "pool_type": "erasure", "erasure_code_profile": "cephfs_data_ec", "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-09T00:04:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:17 smithi073 ceph-mon[32309]: osdmap e87: 12 total, 12 up, 12 in 2024-03-09T00:04:18.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:17 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd=[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]: dispatch 2024-03-09T00:04:18.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.941+0000 7f15eaffd640 1 -- 172.21.15.73:0/1769347879 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]=0 added data pool 4 to fsmap v16) v1 ==== 137+0+0 (secure 0 0 0) 0x7f15f009b950 con 0x7f15f41001f0 2024-03-09T00:04:18.937 INFO:teuthology.orchestra.run.smithi073.stderr:added data pool 4 to fsmap 2024-03-09T00:04:18.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 msgr2=0x7f15e00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:18.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 0x7f15e00a4cc0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f15f41979b0 tx=0x7f15e4005e50 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 msgr2=0x7f15f41964a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f15f000d770 tx=0x7f15f000dc30 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 shutdown_connections 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15e00a2800 0x7f15e00a4cc0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15f4101da0 0x7f15f419da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15f41013e0 0x7f15f41969e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 --2- 172.21.15.73:0/1769347879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15f41001f0 0x7f15f41964a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 >> 172.21.15.73:0/1769347879 conn(0x7f15f40fb7a0 msgr2=0x7f15f41046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 shutdown_connections 2024-03-09T00:04:18.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:18.944+0000 7f15fbaea640 1 -- 172.21.15.73:0/1769347879 wait complete. 2024-03-09T00:04:18.951 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- 172.21.15.73:0/2619647269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7780ff5e0 msgr2=0x7fc7780ffa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 --2- 172.21.15.73:0/2619647269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7780ff5e0 0x7fc7780ffa00 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fc76c009930 tx=0x7fc76c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- 172.21.15.73:0/2619647269 shutdown_connections 2024-03-09T00:04:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 --2- 172.21.15.73:0/2619647269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 0x7fc778109800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 --2- 172.21.15.73:0/2619647269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7781025b0 0x7fc778102a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 --2- 172.21.15.73:0/2619647269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7780ff5e0 0x7fc7780ffa00 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- 172.21.15.73:0/2619647269 >> 172.21.15.73:0/2619647269 conn(0x7fc7780faf10 msgr2=0x7fc7780fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- 172.21.15.73:0/2619647269 shutdown_connections 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- 172.21.15.73:0/2619647269 wait complete. 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 Processor -- start 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.060+0000 7fc77c868640 1 -- start start 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 0x7fc7781976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 0x7fc77819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77810d010 con 0x7fc7781025b0 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77810ce90 con 0x7fc7780ff5e0 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc77810d190 con 0x7fc778102f70 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42220/0 (socket says 172.21.15.73:42220) 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 -- 172.21.15.73:0/470026698 learned_addr learned my addr 172.21.15.73:0/470026698 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc776d9d640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 0x7fc77819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77659c640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 0x7fc7781976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 -- 172.21.15.73:0/470026698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 msgr2=0x7fc77819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 0x7fc77819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 -- 172.21.15.73:0/470026698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 msgr2=0x7fc7781976b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 0x7fc7781976b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 -- 172.21.15.73:0/470026698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc77819f2d0 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77659c640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 0x7fc7781976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc775d9b640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c890 tx=0x7fc76800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc768004220 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7680043c0 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc77819f5c0 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7680049e0 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.061+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc77819fb10 con 0x7fc7781025b0 2024-03-09T00:04:19.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.062+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc778108760 con 0x7fc7781025b0 2024-03-09T00:04:19.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.063+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc7680079b0 con 0x7fc7781025b0 2024-03-09T00:04:19.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.064+0000 7fc7677fe640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 0x7fc7600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.064+0000 7fc77659c640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 0x7fc7600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.064+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc7680cedc0 con 0x7fc7781025b0 2024-03-09T00:04:19.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.064+0000 7fc77659c640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 0x7fc7600a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036040 tx=0x7fc76c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.068+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc768014030 con 0x7fc7781025b0 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: 2024-03-09T00:04:18.960+0000 7f786d93e700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-mon[36106]: pgmap v294: 129 pgs: 10 creating+peering, 89 active+clean, 30 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-mon[36106]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-mon[36106]: osdmap e88: 12 total, 12 up, 12 in 2024-03-09T00:04:19.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:18 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-09T00:04:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.283+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc77802d250 con 0x7fc7781025b0 2024-03-09T00:04:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.284+0000 7fc7677fe640 1 -- 172.21.15.73:0/470026698 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11292 (secure 0 0 0) 0x7fc77802d250 con 0x7fc7781025b0 2024-03-09T00:04:19.279 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:19.280 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:18.298737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-09T00:04:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 16 2024-03-09T00:04:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 msgr2=0x7fc7600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 0x7fc7600a4cc0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fc76c036040 tx=0x7fc76c02cf90 comp rx=0 tx=0).stop 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 msgr2=0x7fc778197bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7fc76800c890 tx=0x7fc76800cd50 comp rx=0 tx=0).stop 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 shutdown_connections 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7600a2800 0x7fc7600a4cc0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc778102f70 0x7fc77819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7781025b0 0x7fc778197bf0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 --2- 172.21.15.73:0/470026698 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7780ff5e0 0x7fc7781976b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 >> 172.21.15.73:0/470026698 conn(0x7fc7780faf10 msgr2=0x7fc778105930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 shutdown_connections 2024-03-09T00:04:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.286+0000 7fc77c868640 1 -- 172.21.15.73:0/470026698 wait complete. 2024-03-09T00:04:19.291 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:18 smithi073 ceph-mon[32309]: pgmap v294: 129 pgs: 10 creating+peering, 89 active+clean, 30 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-09T00:04:19.291 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:18 smithi073 ceph-mon[32309]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:19.292 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:18 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-09T00:04:19.292 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:18 smithi073 ceph-mon[32309]: osdmap e88: 12 total, 12 up, 12 in 2024-03-09T00:04:19.292 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:18 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-09T00:04:19.293 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:18 smithi179 ceph-mon[36637]: pgmap v294: 129 pgs: 10 creating+peering, 89 active+clean, 30 unknown; 577 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s wr, 8 op/s 2024-03-09T00:04:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:18 smithi179 ceph-mon[36637]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-09T00:04:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:18 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1746646070' entity='client.admin' cmd='[{"prefix": "osd pool set", "pool": "cephfs_data_ec", "var": "allow_ec_overwrites", "val": "true"}]': finished 2024-03-09T00:04:19.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:18 smithi179 ceph-mon[36637]: osdmap e88: 12 total, 12 up, 12 in 2024-03-09T00:04:19.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:18 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd=[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]: dispatch 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- 172.21.15.73:0/896810753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101dc0 msgr2=0x7f7fe4108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- 172.21.15.73:0/896810753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101dc0 0x7f7fe4108650 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f7fdc009ca0 tx=0x7f7fdc02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- 172.21.15.73:0/896810753 shutdown_connections 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- 172.21.15.73:0/896810753 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101dc0 0x7f7fe4108650 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- 172.21.15.73:0/896810753 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fe4101400 0x7f7fe4101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- 172.21.15.73:0/896810753 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fe4100210 0x7f7fe4100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- 172.21.15.73:0/896810753 >> 172.21.15.73:0/896810753 conn(0x7f7fe40fb780 msgr2=0x7f7fe40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- 172.21.15.73:0/896810753 shutdown_connections 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- 172.21.15.73:0/896810753 wait complete. 2024-03-09T00:04:19.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 Processor -- start 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 -- start start 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fe4100210 0x7f7fe41965c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.400+0000 7f7fec497640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fe4101dc0 0x7f7fe419db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fec497640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe410bf80 con 0x7f7fe4101400 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fec497640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe410be00 con 0x7f7fe4100210 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fec497640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe410c100 con 0x7f7fe4101dc0 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42238/0 (socket says 172.21.15.73:42238) 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 -- 172.21.15.73:0/475588495 learned_addr learned my addr 172.21.15.73:0/475588495 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 -- 172.21.15.73:0/475588495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fe4101dc0 msgr2=0x7f7fe419db70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fe4101dc0 0x7f7fe419db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 -- 172.21.15.73:0/475588495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fe4100210 msgr2=0x7f7fe41965c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fe4100210 0x7f7fe41965c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 -- 172.21.15.73:0/475588495 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe419e1e0 con 0x7f7fe4101400 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fe9a32640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000b4a0 tx=0x7f7fe000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0014070 con 0x7f7fe4101400 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fe0010af0 con 0x7f7fe4101400 2024-03-09T00:04:19.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe419e4d0 con 0x7f7fe4101400 2024-03-09T00:04:19.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7fe0017d10 con 0x7f7fe4101400 2024-03-09T00:04:19.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.401+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe419eaa0 con 0x7f7fe4101400 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.402+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe41074d0 con 0x7f7fe4101400 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.403+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7fe0017070 con 0x7f7fe4101400 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.404+0000 7f7fdb7fe640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 0x7f7fd00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.404+0000 7f7fea233640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 0x7f7fd00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.404+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f7fe00ce0c0 con 0x7f7fe4101400 2024-03-09T00:04:19.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.404+0000 7f7fea233640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 0x7f7fd00a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4101260 tx=0x7f7fd4006c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.409+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7fe009af20 con 0x7f7fe4101400 2024-03-09T00:04:19.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.582+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7fe4061f90 con 0x7f7fe4101400 2024-03-09T00:04:19.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.584+0000 7f7fdb7fe640 1 -- 172.21.15.73:0/475588495 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7f7fe009a820 con 0x7f7fe4101400 2024-03-09T00:04:19.579 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:19.579 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 msgr2=0x7f7fd00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 0x7f7fd00a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f7fe4101260 tx=0x7f7fd4006c90 comp rx=0 tx=0).stop 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 msgr2=0x7f7fe4196b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f7fe000b4a0 tx=0x7f7fe000b960 comp rx=0 tx=0).stop 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 shutdown_connections 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7fd00a2800 0x7f7fd00a4cc0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7fe4101dc0 0x7f7fe419db70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7fe4101400 0x7f7fe4196b00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 --2- 172.21.15.73:0/475588495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7fe4100210 0x7f7fe41965c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 >> 172.21.15.73:0/475588495 conn(0x7f7fe40fb780 msgr2=0x7f7fe41046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 shutdown_connections 2024-03-09T00:04:19.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.586+0000 7f7fec497640 1 -- 172.21.15.73:0/475588495 wait complete. 2024-03-09T00:04:19.593 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 -- 172.21.15.73:0/2058164796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 msgr2=0x7feae4100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 --2- 172.21.15.73:0/2058164796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae4100630 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7feadc009ca0 tx=0x7feadc02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 -- 172.21.15.73:0/2058164796 shutdown_connections 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 --2- 172.21.15.73:0/2058164796 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feae4101dc0 0x7feae4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 --2- 172.21.15.73:0/2058164796 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101400 0x7feae4101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 --2- 172.21.15.73:0/2058164796 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae4100630 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 -- 172.21.15.73:0/2058164796 >> 172.21.15.73:0/2058164796 conn(0x7feae40fb780 msgr2=0x7feae40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 -- 172.21.15.73:0/2058164796 shutdown_connections 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.698+0000 7feae902b640 1 -- 172.21.15.73:0/2058164796 wait complete. 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 Processor -- start 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 -- start start 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feae4101400 0x7feae4197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 0x7feae419eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae410be70 con 0x7feae4100210 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae410bcf0 con 0x7feae4101dc0 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.699+0000 7feae902b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae410bff0 con 0x7feae4101400 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42246/0 (socket says 172.21.15.73:42246) 2024-03-09T00:04:19.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 -- 172.21.15.73:0/501609926 learned_addr learned my addr 172.21.15.73:0/501609926 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae359e640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 0x7feae419eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 -- 172.21.15.73:0/501609926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feae4101400 msgr2=0x7feae4197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feae4101400 0x7feae4197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 -- 172.21.15.73:0/501609926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 msgr2=0x7feae419eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 0x7feae419eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 -- 172.21.15.73:0/501609926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feae419f200 con 0x7feae4100210 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae359e640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 0x7feae419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae2d9d640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feadc037040 tx=0x7feadc035b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc039040 con 0x7feae4100210 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feadc044070 con 0x7feae4100210 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae419f490 con 0x7feae4100210 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feadc0044d0 con 0x7feae4100210 2024-03-09T00:04:19.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.700+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae419f980 con 0x7feae4100210 2024-03-09T00:04:19.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.701+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feae419fbe0 con 0x7feae4100210 2024-03-09T00:04:19.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.702+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7feadc003b90 con 0x7feae4100210 2024-03-09T00:04:19.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.703+0000 7feadbfff640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 0x7fead00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:19.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.703+0000 7feae259c640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 0x7fead00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:19.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.703+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7feadc0f13b0 con 0x7feae4100210 2024-03-09T00:04:19.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.703+0000 7feae259c640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 0x7fead00a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feae4198af0 tx=0x7feacc005bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:19.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.707+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feadc0be210 con 0x7feae4100210 2024-03-09T00:04:19.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.883+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feae41a0380 con 0x7feae4100210 2024-03-09T00:04:19.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.884+0000 7feadbfff640 1 -- 172.21.15.73:0/501609926 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7feadc0098b0 con 0x7feae4100210 2024-03-09T00:04:19.879 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:19.880 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:19.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 msgr2=0x7fead00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 0x7fead00a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7feae4198af0 tx=0x7feacc005bf0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 msgr2=0x7feae41975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7feadc037040 tx=0x7feadc035b60 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 shutdown_connections 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fead00a2800 0x7fead00a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feae4101dc0 0x7feae419eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feae4101400 0x7feae4197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 --2- 172.21.15.73:0/501609926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feae4100210 0x7feae41975e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.886+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 >> 172.21.15.73:0/501609926 conn(0x7feae40fb780 msgr2=0x7feae40fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.887+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 shutdown_connections 2024-03-09T00:04:19.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:19.887+0000 7feae902b640 1 -- 172.21.15.73:0/501609926 wait complete. 2024-03-09T00:04:19.912 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:20.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 -- 172.21.15.73:0/4266570404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b60100500 msgr2=0x7f1b60100980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 --2- 172.21.15.73:0/4266570404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b60100500 0x7f1b60100980 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f1b50009900 tx=0x7f1b5002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 -- 172.21.15.73:0/4266570404 shutdown_connections 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 --2- 172.21.15.73:0/4266570404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 0x7f1b60108760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 --2- 172.21.15.73:0/4266570404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b60100500 0x7f1b60100980 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 --2- 172.21.15.73:0/4266570404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b601017d0 0x7f1b60101bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 -- 172.21.15.73:0/4266570404 >> 172.21.15.73:0/4266570404 conn(0x7f1b600faf30 msgr2=0x7f1b600fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 -- 172.21.15.73:0/4266570404 shutdown_connections 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.019+0000 7f1b674a4640 1 -- 172.21.15.73:0/4266570404 wait complete. 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 Processor -- start 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 -- start start 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 0x7f1b601976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 0x7f1b60197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6010bee0 con 0x7f1b601017d0 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6010bd60 con 0x7f1b60100ec0 2024-03-09T00:04:20.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b674a4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6010c060 con 0x7f1b60100500 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42268/0 (socket says 172.21.15.73:42268) 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 -- 172.21.15.73:0/499362527 learned_addr learned my addr 172.21.15.73:0/499362527 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b64a3f640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 0x7f1b60197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65240640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 0x7f1b601976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 -- 172.21.15.73:0/499362527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 msgr2=0x7f1b601976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 0x7f1b601976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 -- 172.21.15.73:0/499362527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 msgr2=0x7f1b60197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 0x7f1b60197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 -- 172.21.15.73:0/499362527 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6019f2c0 con 0x7f1b601017d0 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65240640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 0x7f1b601976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.020+0000 7f1b65a41640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800cad0 tx=0x7f1b5800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.021+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800ae20 con 0x7f1b601017d0 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.021+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b58005020 con 0x7f1b601017d0 2024-03-09T00:04:20.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.021+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1b5800f6f0 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.021+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6019f5b0 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.021+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6019fa80 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.022+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6019fd40 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.023+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1b580051d0 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.023+0000 7f1b567fc640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 0x7f1b4c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.023+0000 7f1b65240640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 0x7f1b4c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.023+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f1b580ced40 con 0x7f1b601017d0 2024-03-09T00:04:20.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.024+0000 7f1b65240640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 0x7f1b4c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c005d00 tx=0x7f1b5c005ed0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.027+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1b58014030 con 0x7f1b601017d0 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: osdmap e89: 12 total, 12 up, 12 in 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/470026698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/475588495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/501609926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.241+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1b6019ffb0 con 0x7f1b601017d0 2024-03-09T00:04:20.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.242+0000 7f1b567fc640 1 -- 172.21.15.73:0/499362527 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+11292 (secure 0 0 0) 0x7f1b5809bba0 con 0x7f1b601017d0 2024-03-09T00:04:20.237 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:20.237 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24493,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[{"mdsmap":{"epoch":16,"flags":18,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":false,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:18.298737+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":1},"id":1}]} 2024-03-09T00:04:20.237 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 16 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 msgr2=0x7f1b4c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 0x7f1b4c0a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1b5c005d00 tx=0x7f1b5c005ed0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 msgr2=0x7f1b6019ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f1b5800cad0 tx=0x7f1b5800cf90 comp rx=0 tx=0).stop 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 shutdown_connections 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1b4c0a2800 0x7f1b4c0a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1b601017d0 0x7f1b6019ec50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1b60100ec0 0x7f1b60197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 --2- 172.21.15.73:0/499362527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1b60100500 0x7f1b601976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 >> 172.21.15.73:0/499362527 conn(0x7f1b600faf30 msgr2=0x7f1b600ff3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 shutdown_connections 2024-03-09T00:04:20.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.244+0000 7f1b674a4640 1 -- 172.21.15.73:0/499362527 wait complete. 2024-03-09T00:04:20.250 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: osdmap e89: 12 total, 12 up, 12 in 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/470026698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/475588495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/501609926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: osdmap e89: 12 total, 12 up, 12 in 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1769347879' entity='client.admin' cmd='[{"prefix": "fs add_data_pool", "fs_name": "cephfs", "pool": "cephfs_data_ec"}]': finished 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/470026698' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/475588495' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/501609926' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 -- 172.21.15.73:0/2425032431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc100230 msgr2=0x7f2cdc100650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/2425032431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc100230 0x7f2cdc100650 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc009900 tx=0x7f2ccc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 -- 172.21.15.73:0/2425032431 shutdown_connections 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/2425032431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc101de0 0x7f2cdc108670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/2425032431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc101420 0x7f2cdc1018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/2425032431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc100230 0x7f2cdc100650 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 -- 172.21.15.73:0/2425032431 >> 172.21.15.73:0/2425032431 conn(0x7f2cdc0fb7a0 msgr2=0x7f2cdc0fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 -- 172.21.15.73:0/2425032431 shutdown_connections 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.356+0000 7f2ce15b9640 1 -- 172.21.15.73:0/2425032431 wait complete. 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 Processor -- start 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 -- start start 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc100230 0x7f2cdc06a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc101de0 0x7f2cdc19e900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc10bec0 con 0x7f2cdc101420 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc10bd40 con 0x7f2cdc101de0 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2ce15b9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cdc10c040 con 0x7f2cdc100230 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42288/0 (socket says 172.21.15.73:42288) 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 -- 172.21.15.73:0/1226459055 learned_addr learned my addr 172.21.15.73:0/1226459055 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cdaffd640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc100230 0x7f2cdc06a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 -- 172.21.15.73:0/1226459055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc100230 msgr2=0x7f2cdc06a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc100230 0x7f2cdc06a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 -- 172.21.15.73:0/1226459055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc101de0 msgr2=0x7f2cdc19e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc101de0 0x7f2cdc19e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.357+0000 7f2cd3fff640 1 -- 172.21.15.73:0/1226459055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cdc19f1c0 con 0x7f2cdc101420 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2cd3fff640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c920 tx=0x7f2cc400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4007c70 con 0x7f2cdc101420 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2cc4007e10 con 0x7f2cdc101420 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cdc19f4b0 con 0x7f2cdc101420 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2cc4018890 con 0x7f2cdc101420 2024-03-09T00:04:20.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.358+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cdc19fa00 con 0x7f2cdc101420 2024-03-09T00:04:20.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.359+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cdc107550 con 0x7f2cdc101420 2024-03-09T00:04:20.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.360+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2cc4018a70 con 0x7f2cdc101420 2024-03-09T00:04:20.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.360+0000 7f2cd8ff9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 0x7f2cc80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.360+0000 7f2cdaffd640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 0x7f2cc80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.360+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f2cc40cee70 con 0x7f2cdc101420 2024-03-09T00:04:20.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.361+0000 7f2cdaffd640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 0x7f2cc80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc02c7b0 tx=0x7f2ccc005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.364+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2cc4014030 con 0x7f2cdc101420 2024-03-09T00:04:20.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.539+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2cdc061f90 con 0x7f2cdc101420 2024-03-09T00:04:20.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.541+0000 7f2cd8ff9640 1 -- 172.21.15.73:0/1226459055 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7f2cc400de20 con 0x7f2cdc101420 2024-03-09T00:04:20.536 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:20.536 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 msgr2=0x7f2cc80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 0x7f2cc80a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f2ccc02c7b0 tx=0x7f2ccc005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 msgr2=0x7f2cdc19c2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f2cc400c920 tx=0x7f2cc400cde0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 shutdown_connections 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2cc80a2800 0x7f2cc80a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2cdc101de0 0x7f2cdc19e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2cdc101420 0x7f2cdc19c2c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 --2- 172.21.15.73:0/1226459055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2cdc100230 0x7f2cdc06a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 >> 172.21.15.73:0/1226459055 conn(0x7f2cdc0fb7a0 msgr2=0x7f2cdc104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 shutdown_connections 2024-03-09T00:04:20.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.543+0000 7f2ce15b9640 1 -- 172.21.15.73:0/1226459055 wait complete. 2024-03-09T00:04:20.549 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 16, 'flags': 18, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': False, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2024-03-09T00:04:13.661086+0000', 'modified': '2024-03-09T00:04:18.298737+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6839/3264474111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 3264474111}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 3264474111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 1} 2024-03-09T00:04:20.550 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-09T00:04:20.550 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-09T00:04:20.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 -- 172.21.15.73:0/479240495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ffd40 msgr2=0x7f26ac1098f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 --2- 172.21.15.73:0/479240495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ffd40 0x7f26ac1098f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009ca0 tx=0x7f26a402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 -- 172.21.15.73:0/479240495 shutdown_connections 2024-03-09T00:04:20.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 --2- 172.21.15.73:0/479240495 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ffd40 0x7f26ac1098f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 --2- 172.21.15.73:0/479240495 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26ac0ff380 0x7f26ac0ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 --2- 172.21.15.73:0/479240495 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac102910 0x7f26ac102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 -- 172.21.15.73:0/479240495 >> 172.21.15.73:0/479240495 conn(0x7f26ac0faee0 msgr2=0x7f26ac0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 -- 172.21.15.73:0/479240495 shutdown_connections 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 -- 172.21.15.73:0/479240495 wait complete. 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.657+0000 7f26b30c0640 1 Processor -- start 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 -- start start 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac0ffd40 0x7f26ac197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26ac102910 0x7f26ac19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac10cff0 con 0x7f26ac0ff380 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac10ce70 con 0x7f26ac102910 2024-03-09T00:04:20.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26ac10d170 con 0x7f26ac0ffd40 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42310/0 (socket says 172.21.15.73:42310) 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 -- 172.21.15.73:0/3545960246 learned_addr learned my addr 172.21.15.73:0/3545960246 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26a3fff640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac0ffd40 0x7f26ac197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 -- 172.21.15.73:0/3545960246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac0ffd40 msgr2=0x7f26ac197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac0ffd40 0x7f26ac197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 -- 172.21.15.73:0/3545960246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26ac102910 msgr2=0x7f26ac19eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26ac102910 0x7f26ac19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 -- 172.21.15.73:0/3545960246 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26ac19f1f0 con 0x7f26ac0ff380 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f26a800e9c0 tx=0x7f26a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26a1ffb640 1 -- 172.21.15.73:0/3545960246 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a800cd80 con 0x7f26ac0ff380 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26a1ffb640 1 -- 172.21.15.73:0/3545960246 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a80035f0 con 0x7f26ac0ff380 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26ac19f4e0 con 0x7f26ac0ff380 2024-03-09T00:04:20.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.658+0000 7f26a1ffb640 1 -- 172.21.15.73:0/3545960246 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8010690 con 0x7f26ac0ff380 2024-03-09T00:04:20.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.659+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26ac19f9e0 con 0x7f26ac0ff380 2024-03-09T00:04:20.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.660+0000 7f26a1ffb640 1 -- 172.21.15.73:0/3545960246 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f26a80037a0 con 0x7f26ac0ff380 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 Processor -- start 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 -- start start 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a1ffb640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 0x7f26940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787940c500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557879411c40 0x5578794140d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5578793717c0 con 0x55787940cad0 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787931ab50 con 0x557879411c40 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787936d860 con 0x55787940c0e0 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b165d640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 0x7f26940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a1ffb640 1 -- 172.21.15.73:0/3545960246 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f26a80cf0c0 con 0x7f26ac0ff380 2024-03-09T00:04:20.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42316/0 (socket says 172.21.15.73:42316) 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 -- 172.21.15.73:0/3324602281 learned_addr learned my addr 172.21.15.73:0/3324602281 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787940c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26b165d640 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557879411c40 0x5578794140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 -- 172.21.15.73:0/3324602281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 msgr2=0x55787940c500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787940c500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 -- 172.21.15.73:0/3324602281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557879411c40 msgr2=0x5578794140d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557879411c40 0x5578794140d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.661+0000 7f26a3fff640 1 -- 172.21.15.73:0/3324602281 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55787932e9a0 con 0x55787940cad0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b165d640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 0x7f26940a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006010 tx=0x7f26a4032d10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26a3fff640 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f269c00bce0 tx=0x7f269c00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8953945b77160d5b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f2692ffd640 1 -- 172.21.15.73:0/3324602281 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c015070 con 0x55787940cad0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f2692ffd640 1 -- 172.21.15.73:0/3324602281 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269c01a940 con 0x55787940cad0 2024-03-09T00:04:20.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f2692ffd640 1 -- 172.21.15.73:0/3324602281 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c0118f0 con 0x55787940cad0 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 -- 172.21.15.73:0/3324602281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 msgr2=0x5578794115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f269c00bce0 tx=0x7f269c00f730 comp rx=0 tx=0).stop 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 -- 172.21.15.73:0/3324602281 shutdown_connections 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557879411c40 0x5578794140d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940cad0 0x5578794115d0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/3324602281 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787940c500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.662+0000 7f26b5d7b740 1 -- 172.21.15.73:0/3324602281 >> 172.21.15.73:0/3324602281 conn(0x5578793d4400 msgr2=0x557879407a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- 172.21.15.73:0/3324602281 shutdown_connections 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- 172.21.15.73:0/3324602281 wait complete. 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- start start 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 Processor -- start 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940cad0 0x557879416ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879414bf0 con 0x557879411c40 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55787932df90 con 0x55787940c0e0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879414d70 con 0x55787940cad0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b165d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b5d7b740 10 client.0 _ll_register_callbacks cb 0x557879369400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b165d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42324/0 (socket says 172.21.15.73:42324) 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26b165d640 1 -- 172.21.15.73:0/797291018 learned_addr learned my addr 172.21.15.73:0/797291018 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.663+0000 7f26a3fff640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940cad0 0x557879416ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 -- 172.21.15.73:0/797291018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940cad0 msgr2=0x557879416ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940cad0 0x557879416ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 -- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940c0e0 msgr2=0x557879416590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 -- 172.21.15.73:0/797291018 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578790862d0 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b165d640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009ca0 tx=0x7f26a4042000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=856e962d28d2c46b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4051060 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f26a4004450 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f269c01a940 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26a4058050 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 10 client.14766 fetch_fsmap learned FSMap version 16 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x5578791506b0 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269c01a940 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 16) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f26a404b070 con 0x557879411c40 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 10 client.14766 fetch_fsmap finished waiting for FSMap version 16 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 10 client.14766 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 20 client.14766 populate_metadata read hostname 'smithi073' 2024-03-09T00:04:20.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557879423160 0x5578794221e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 4 client.14766 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.664+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55787937ed00 con 0x557879423160 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.665+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557879423160 0x5578794221e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.665+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557879423160 0x5578794221e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=40ea569b2e0fa2fd server_cookie=5d445deef1d5da5a in_seq=0 out_seq=0 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.665+0000 7f268bfff640 10 client.14766 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f268bfff640 1 -- 172.21.15.73:0/797291018 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55787937ed00 con 0x557879423160 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f268bfff640 10 client.14766 handle_command_reply: tid=0 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 client.14766 shutdown 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 msgr2=0x55787941db10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f26a4009ca0 tx=0x7f26a4042000 comp rx=0 tx=0).stop 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 shutdown_connections 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557879423160 0x5578794221e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x55787941db10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940cad0 0x557879416ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f268bfff640 0 client.14766 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/797291018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 >> 172.21.15.73:0/797291018 conn(0x5578793d4400 msgr2=0x557879406c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 shutdown_connections 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.666+0000 7f26b5d7b740 1 -- 172.21.15.73:0/797291018 wait complete. 2024-03-09T00:04:20.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.667+0000 7f26b5d7b740 20 client.14766 trim_cache size 0 max 16384 2024-03-09T00:04:20.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 Processor -- start 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 -- start start 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941dad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557879411c40 0x557879416ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879422980 con 0x55787940c0e0 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879422fe0 con 0x55787940cad0 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879423640 con 0x557879411c40 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42340/0 (socket says 172.21.15.73:42340) 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 -- 172.21.15.73:0/1233988894 learned_addr learned my addr 172.21.15.73:0/1233988894 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b165d640 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557879411c40 0x557879416ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 -- 172.21.15.73:0/1233988894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557879411c40 msgr2=0x557879416ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557879411c40 0x557879416ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 -- 172.21.15.73:0/1233988894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 msgr2=0x55787941dad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 -- 172.21.15.73:0/1233988894 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55787942be90 con 0x55787940c0e0 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.682+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f26a80058c0 tx=0x7f26a8097480 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a100f43a0764d2c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26927fc640 1 -- 172.21.15.73:0/1233988894 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a8078cb0 con 0x55787940c0e0 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26927fc640 1 -- 172.21.15.73:0/1233988894 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a809bf20 con 0x55787940c0e0 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26927fc640 1 -- 172.21.15.73:0/1233988894 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a80998c0 con 0x55787940c0e0 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- 172.21.15.73:0/1233988894 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 msgr2=0x557879416590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f26a80058c0 tx=0x7f26a8097480 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- 172.21.15.73:0/1233988894 shutdown_connections 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557879411c40 0x557879416ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941dad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/1233988894 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55787940c0e0 0x557879416590 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- 172.21.15.73:0/1233988894 >> 172.21.15.73:0/1233988894 conn(0x5578793d4400 msgr2=0x55787940df30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- 172.21.15.73:0/1233988894 shutdown_connections 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- 172.21.15.73:0/1233988894 wait complete. 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 -- start start 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.683+0000 7f26b5d7b740 1 Processor -- start 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787941a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879424300 con 0x557879411c40 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879423ca0 con 0x55787940cad0 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557879424960 con 0x55787940c0e0 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 10 client.0 _ll_register_callbacks cb 0x7f269c00b480 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42342/0 (socket says 172.21.15.73:42342) 2024-03-09T00:04:20.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 -- 172.21.15.73:0/2175601769 learned_addr learned my addr 172.21.15.73:0/2175601769 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26a3fff640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 -- 172.21.15.73:0/2175601769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 msgr2=0x55787941a8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787941a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 -- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 msgr2=0x55787941ae00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 -- 172.21.15.73:0/2175601769 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5578794792a0 con 0x557879411c40 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b165d640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f26a40093c0 tx=0x7f26a4042650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f847cb4681e974 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f268bfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f26a4051040 con 0x557879411c40 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f26a4004660 con 0x557879411c40 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.684+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f26a809bf20 con 0x557879411c40 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=16) v2 ==== 24+0+0 (secure 0 0 0) 0x7f26a4034440 con 0x557879411c40 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 10 client.14772 fetch_fsmap learned FSMap version 16 2024-03-09T00:04:20.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=16,monmap=4+}) v3 -- 0x55787947a450 con 0x557879411c40 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f26a809bf20 con 0x557879411c40 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 16) v1 ==== 7902+0+0 (secure 0 0 0) 0x7f26a404b040 con 0x557879411c40 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 10 client.14772 fetch_fsmap finished waiting for FSMap version 16 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 10 client.14772 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 20 client.14772 populate_metadata read hostname 'smithi073' 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55787947ebb0 0x55787947a6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 4 client.14772 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "status"}) v1 -- 0x557879379050 con 0x55787947ebb0 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.685+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55787947ebb0 0x55787947a6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f268bfff640 10 client.14772 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b0e5c640 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55787947ebb0 0x55787947a6e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=85f3d005fbef88ae server_cookie=cbd1a968344e919e in_seq=0 out_seq=0 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f268bfff640 1 -- 172.21.15.73:0/2175601769 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+325 (crc 0 0 0) 0x557879379050 con 0x55787947ebb0 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f268bfff640 10 client.14772 handle_command_reply: tid=0 2024-03-09T00:04:20.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 client.14772 shutdown 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 msgr2=0x557879478d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f26a40093c0 tx=0x7f26a4042650 comp rx=0 tx=0).stop 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 shutdown_connections 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55787947ebb0 0x55787947a6e0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557879411c40 0x557879478d60 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f268bfff640 0 client.14772 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55787940cad0 0x55787941ae00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 --2- 172.21.15.73:0/2175601769 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55787940c0e0 0x55787941a8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 >> 172.21.15.73:0/2175601769 conn(0x5578793d4400 msgr2=0x557879416590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 shutdown_connections 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 1 -- 172.21.15.73:0/2175601769 wait complete. 2024-03-09T00:04:20.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.686+0000 7f26b5d7b740 20 client.14772 trim_cache size 0 max 16384 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "cluster_fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "whoami": 0, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "id": 24515, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "want_state": "up:active", 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "state": "up:active", 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "fs_name": "cephfs", 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "rank_uptime": 6.3649292859999997, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "mdsmap_epoch": 16, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch": 89, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch_barrier": 89, 2024-03-09T00:04:20.689 INFO:teuthology.orchestra.run.smithi073.stdout: "uptime": 11.77142516 2024-03-09T00:04:20.690 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:04:20.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 msgr2=0x7f26940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 0x7f26940a4cc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f26a4006010 tx=0x7f26a4032d10 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 msgr2=0x7f26ac1975d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f26a800e9c0 tx=0x7f26a800ee80 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 shutdown_connections 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f26940a2800 0x7f26940a4cc0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26ac102910 0x7f26ac19eb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26ac0ffd40 0x7f26ac197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 --2- 172.21.15.73:0/3545960246 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26ac0ff380 0x7f26ac1975d0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 >> 172.21.15.73:0/3545960246 conn(0x7f26ac0faee0 msgr2=0x7f26ac1059a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 shutdown_connections 2024-03-09T00:04:20.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:20.687+0000 7f26b30c0640 1 -- 172.21.15.73:0/3545960246 wait complete. 2024-03-09T00:04:20.693 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-09T00:04:20.693 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:20.693 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-09T00:04:20.929 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:21.201 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:20 smithi073 ceph-mon[32309]: pgmap v297: 129 pgs: 2 creating+peering, 107 active+clean, 20 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-09T00:04:21.201 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:20 smithi073 ceph-mon[32309]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-09T00:04:21.201 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:20 smithi073 ceph-mon[32309]: Cluster is now healthy 2024-03-09T00:04:21.201 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:20 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/499362527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:21.202 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:20 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1226459055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:20 smithi102 ceph-mon[36106]: pgmap v297: 129 pgs: 2 creating+peering, 107 active+clean, 20 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-09T00:04:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:20 smithi102 ceph-mon[36106]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-09T00:04:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:20 smithi102 ceph-mon[36106]: Cluster is now healthy 2024-03-09T00:04:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:20 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/499362527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:21.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:20 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1226459055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:21.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:20 smithi179 ceph-mon[36637]: pgmap v297: 129 pgs: 2 creating+peering, 107 active+clean, 20 unknown; 579 KiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s wr, 8 op/s 2024-03-09T00:04:21.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:20 smithi179 ceph-mon[36637]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-09T00:04:21.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:20 smithi179 ceph-mon[36637]: Cluster is now healthy 2024-03-09T00:04:21.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:20 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/499362527' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:21.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:20 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1226459055' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 -- 172.21.15.73:0/1399429346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 msgr2=0x7f95e0106a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/1399429346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0106a40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f95cc0099c0 tx=0x7f95cc009cd0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 -- 172.21.15.73:0/1399429346 shutdown_connections 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/1399429346 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 0x7f95e010ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/1399429346 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 0x7f95e01024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/1399429346 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0106a40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.705+0000 7f95e6bb7700 1 -- 172.21.15.73:0/1399429346 >> 172.21.15.73:0/1399429346 conn(0x7f95e0078f00 msgr2=0x7f95e0079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- 172.21.15.73:0/1399429346 shutdown_connections 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- 172.21.15.73:0/1399429346 wait complete. 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 Processor -- start 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- start start 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 0x7f95e0198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 0x7f95e019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e0112140 con 0x7f95e0102a70 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e0111fc0 con 0x7f95e0107170 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95e6bb7700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95e01122c0 con 0x7f95e01020c0 2024-03-09T00:04:22.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.706+0000 7f95dffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42356/0 (socket says 172.21.15.73:42356) 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 -- 172.21.15.73:0/2829226843 learned_addr learned my addr 172.21.15.73:0/2829226843 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95e5154700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 0x7f95e019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95e4953700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 0x7f95e0198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 -- 172.21.15.73:0/2829226843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 msgr2=0x7f95e0198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 0x7f95e0198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 -- 172.21.15.73:0/2829226843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 msgr2=0x7f95e019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 0x7f95e019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 -- 172.21.15.73:0/2829226843 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95e019d5a0 con 0x7f95e0102a70 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95dffff700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009fd0 tx=0x7f95cc005f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc027070 con 0x7f95e0102a70 2024-03-09T00:04:22.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95e019d830 con 0x7f95e0102a70 2024-03-09T00:04:22.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95cc004050 con 0x7f95e0102a70 2024-03-09T00:04:22.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95cc022e00 con 0x7f95e0102a70 2024-03-09T00:04:22.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.707+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95e01a5110 con 0x7f95e0102a70 2024-03-09T00:04:22.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.708+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95e010a320 con 0x7f95e0102a70 2024-03-09T00:04:22.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.709+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f95cc02a070 con 0x7f95e0102a70 2024-03-09T00:04:22.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.710+0000 7f95ddffb700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 0x7f95d00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:22.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.710+0000 7f95e4953700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 0x7f95d00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:22.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.710+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f95cc0cff60 con 0x7f95e0102a70 2024-03-09T00:04:22.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.711+0000 7f95e4953700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 0x7f95d00a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006fd0 tx=0x7f95d4008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:22.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.714+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f95cc013070 con 0x7f95e0102a70 2024-03-09T00:04:22.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.958+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f95e01998f0 con 0x7f95e0102a70 2024-03-09T00:04:22.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.958+0000 7f95ddffb700 1 -- 172.21.15.73:0/2829226843 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+88 (secure 0 0 0) 0x7f95cc09ce50 con 0x7f95e0102a70 2024-03-09T00:04:22.953 INFO:teuthology.orchestra.run.smithi073.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-09T00:04:22.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.960+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 msgr2=0x7f95d00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.960+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 0x7f95d00a4dd0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f95d4006fd0 tx=0x7f95d4008040 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.960+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 msgr2=0x7f95e0198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.960+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f95cc009fd0 tx=0x7f95cc005f10 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 shutdown_connections 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f95d00a2910 0x7f95d00a4dd0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f95e0107170 0x7f95e019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f95e01020c0 0x7f95e0198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 --2- 172.21.15.73:0/2829226843 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f95e0102a70 0x7f95e0198ad0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 >> 172.21.15.73:0/2829226843 conn(0x7f95e0078f00 msgr2=0x7f95e0101600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:22.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 shutdown_connections 2024-03-09T00:04:22.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:22.961+0000 7f95e6bb7700 1 -- 172.21.15.73:0/2829226843 wait complete. 2024-03-09T00:04:23.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:22 smithi102 ceph-mon[36106]: pgmap v298: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-03-09T00:04:23.331 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:22 smithi073 ceph-mon[32309]: pgmap v298: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-03-09T00:04:23.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:22 smithi179 ceph-mon[36637]: pgmap v298: 129 pgs: 2 creating+peering, 127 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s wr, 5 op/s 2024-03-09T00:04:23.677 INFO:tasks.cephfs.mount:Mounting Ceph FS cephfs; just confirmed its presence on cluster 2024-03-09T00:04:23.677 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-09T00:04:23.677 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi073.front.sepia.ceph.com 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:self.client.name = client.admin 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:self.cephfs_name = cephfs 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-09T00:04:23.678 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.admin' 2024-03-09T00:04:23.679 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:23.679 DEBUG:teuthology.orchestra.run.smithi073:> ip addr 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: inet 127.0.0.1/8 scope host lo 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 ::1/128 scope host 2024-03-09T00:04:23.696 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:ea brd ff:ff:ff:ff:ff:ff 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f0 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:eb brd ff:ff:ff:ff:ff:ff 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f1 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:04:23.697 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7c brd ff:ff:ff:ff:ff:ff 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7d brd ff:ff:ff:ff:ff:ff 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout: inet 172.21.15.73/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft 42214sec preferred_lft 42214sec 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-03-09T00:04:23.698 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:04:23.699 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-09T00:04:23.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:23.699 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add name ceph-brx type bridge 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr flush dev ceph-brx 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set ceph-brx up 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-09T00:04:23.700 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:04:23.879 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:23.879 DEBUG:teuthology.orchestra.run.smithi073:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-09T00:04:23.906 INFO:teuthology.orchestra.run.smithi073.stdout:1 2024-03-09T00:04:23.908 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:23.908 DEBUG:teuthology.orchestra.run.smithi073:> route 2024-03-09T00:04:23.975 INFO:teuthology.orchestra.run.smithi073.stdout:Kernel IP routing table 2024-03-09T00:04:23.975 INFO:teuthology.orchestra.run.smithi073.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-09T00:04:23.975 INFO:teuthology.orchestra.run.smithi073.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-09T00:04:23.976 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-09T00:04:23.976 INFO:teuthology.orchestra.run.smithi073.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-09T00:04:23.976 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-09T00:04:23.977 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:04:24.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:23 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2829226843' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:04:24.242 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.242 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-03-09T00:04:24.258 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.258 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list-id 2024-03-09T00:04:24.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:23 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2829226843' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:04:24.313 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.313 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:04:24.313 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:04:24.313 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:24.313 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.admin 0 2024-03-09T00:04:24.313 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:04:24.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:23 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2829226843' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:04:24.396 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.admin' with 192.168.0.1/16 2024-03-09T00:04:24.397 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.admin type veth peer name brx.0 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set veth0 up 2024-03-09T00:04:24.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip link set lo up 2024-03-09T00:04:24.398 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.admin ip route add default via 192.168.255.254 2024-03-09T00:04:24.398 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:04:24.543 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.543 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:04:24.543 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:04:24.543 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set brx.0 up 2024-03-09T00:04:24.544 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set dev brx.0 master ceph-brx 2024-03-09T00:04:24.544 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:04:24.586 INFO:tasks.cephfs.fuse_mount:Client client.admin config is {} 2024-03-09T00:04:24.586 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-09T00:04:24.586 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -v /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:24.641 INFO:teuthology.orchestra.run.smithi073.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.admin' 2024-03-09T00:04:24.641 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-09T00:04:24.641 DEBUG:teuthology.orchestra.run.smithi073:> chmod 0000 /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:24.696 DEBUG:teuthology.orchestra.run.smithi073:> sudo modprobe fuse 2024-03-09T00:04:24.767 DEBUG:teuthology.orchestra.run.smithi073:> cat /proc/self/mounts | awk '{print $2}' 2024-03-09T00:04:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:/proc 2024-03-09T00:04:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:/sys 2024-03-09T00:04:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:/dev 2024-03-09T00:04:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/security 2024-03-09T00:04:24.822 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/shm 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/pts 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/run 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/cgroup 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/pstore 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/bpf 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/ 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/selinux 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/mqueue 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/debug 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/hugepages 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/tracing 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-09T00:04:24.823 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/fuse/connections 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/config 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-sysctl.service 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/nfs/rpc_pipefs 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/1000 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/ceph 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/0889c4a145c12a50907c1a0d4afe4f0221b348ed14f889f1b506261d72225a99/merged 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/e9e41b01bc05c131322a5a5fd2a51d0a60235a8152940a202827ff8d3c484f0a/merged 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/0 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/44895da299dd973b20b1adea2e679c3ec4e1cf4dfa2cb0e170e1282c0efd3c1f/merged 2024-03-09T00:04:24.824 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/79a89b132bdc71d472a1ad30c79ae489ec9c46f6d2bef5cc19472a71b72278c4/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/78c53b1d02cb3f340042fcbd56c4b602f2640f6aef008a9a9ad0f0ca7e403e1b/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/704c67fdceb66636af911e671db58a07833bd6ae70c9b2207a41baf55dab4bc7/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/14ddeb6bea0b19292e90c34685afd2bc38eede85d0e74ac075eec860908914c8/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/23ee3b59958e2e7669146074e6d6c617cf1c583307d95b06b078b9086a82ec9b/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/fd674dee122da583831750bbc859d3914152eb8b14a044c47cc22047cfb6dbdb/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/f60084817b334ee1ad03a83101902e2e4eff798a91b4e211399d40dd0cec6073/merged 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:24.825 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:24.826 DEBUG:teuthology.orchestra.run.smithi073:> ls /sys/fs/fuse/connections 2024-03-09T00:04:24.877 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2024-03-09T00:04:24.878 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest && exec sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --admin-socket '/var/run/ceph/$cluster-$name.$pid.asok' /home/ubuntu/cephtest/mnt.admin --id admin --client_fs=cephfs) 2024-03-09T00:04:24.921 DEBUG:teuthology.orchestra.run.smithi073:> sudo modprobe fuse 2024-03-09T00:04:24.950 DEBUG:teuthology.orchestra.run.smithi073:> cat /proc/self/mounts | awk '{print $2}' 2024-03-09T00:04:25.004 INFO:teuthology.orchestra.run.smithi073.stdout:/proc 2024-03-09T00:04:25.004 INFO:teuthology.orchestra.run.smithi073.stdout:/sys 2024-03-09T00:04:25.004 INFO:teuthology.orchestra.run.smithi073.stdout:/dev 2024-03-09T00:04:25.004 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/security 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/shm 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/pts 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/run 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/cgroup 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/pstore 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/bpf 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/ 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/selinux 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/mqueue 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/debug 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/hugepages 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/tracing 2024-03-09T00:04:25.005 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/fuse/connections 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/config 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-sysctl.service 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/nfs/rpc_pipefs 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/1000 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/ceph 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/0889c4a145c12a50907c1a0d4afe4f0221b348ed14f889f1b506261d72225a99/merged 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/e9e41b01bc05c131322a5a5fd2a51d0a60235a8152940a202827ff8d3c484f0a/merged 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/0 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:25.006 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/44895da299dd973b20b1adea2e679c3ec4e1cf4dfa2cb0e170e1282c0efd3c1f/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/79a89b132bdc71d472a1ad30c79ae489ec9c46f6d2bef5cc19472a71b72278c4/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/78c53b1d02cb3f340042fcbd56c4b602f2640f6aef008a9a9ad0f0ca7e403e1b/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/704c67fdceb66636af911e671db58a07833bd6ae70c9b2207a41baf55dab4bc7/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/14ddeb6bea0b19292e90c34685afd2bc38eede85d0e74ac075eec860908914c8/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/23ee3b59958e2e7669146074e6d6c617cf1c583307d95b06b078b9086a82ec9b/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/fd674dee122da583831750bbc859d3914152eb8b14a044c47cc22047cfb6dbdb/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/f60084817b334ee1ad03a83101902e2e4eff798a91b4e211399d40dd0cec6073/merged 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:25.007 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:25.008 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:25.008 DEBUG:teuthology.orchestra.run.smithi073:> ls /sys/fs/fuse/connections 2024-03-09T00:04:25.070 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi073.stderr:2024-03-09T00:04:25.076+0000 7f83f103a480 -1 init, newargv = 0x55869b4da8c0 newargc=15 2024-03-09T00:04:25.071 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi073.stderr:ceph-fuse[76724]: starting ceph client 2024-03-09T00:04:25.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi073.stderr:ceph-fuse[76724]: starting fuse 2024-03-09T00:04:25.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:24 smithi073 ceph-mon[32309]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 732 B/s wr, 4 op/s 2024-03-09T00:04:25.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:24 smithi102 ceph-mon[36106]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 732 B/s wr, 4 op/s 2024-03-09T00:04:25.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:24 smithi179 ceph-mon[36637]: pgmap v299: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 732 B/s wr, 4 op/s 2024-03-09T00:04:26.164 DEBUG:teuthology.orchestra.run.smithi073:> sudo modprobe fuse 2024-03-09T00:04:26.199 DEBUG:teuthology.orchestra.run.smithi073:> cat /proc/self/mounts | awk '{print $2}' 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/proc 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/sys 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/dev 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/security 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/shm 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/pts 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/run 2024-03-09T00:04:26.256 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/cgroup 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/pstore 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/bpf 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/ 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/selinux 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/mqueue 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/debug 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/dev/hugepages 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/tracing 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup-dev.service 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/fs/fuse/connections 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/sys/kernel/config 2024-03-09T00:04:26.257 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-sysctl.service 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/run/credentials/systemd-tmpfiles-setup.service 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/nfs/rpc_pipefs 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/1000 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/ceph 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/0889c4a145c12a50907c1a0d4afe4f0221b348ed14f889f1b506261d72225a99/merged 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/e9e41b01bc05c131322a5a5fd2a51d0a60235a8152940a202827ff8d3c484f0a/merged 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/run/user/0 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/proc/sys/fs/binfmt_misc 2024-03-09T00:04:26.258 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/44895da299dd973b20b1adea2e679c3ec4e1cf4dfa2cb0e170e1282c0efd3c1f/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/79a89b132bdc71d472a1ad30c79ae489ec9c46f6d2bef5cc19472a71b72278c4/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/78c53b1d02cb3f340042fcbd56c4b602f2640f6aef008a9a9ad0f0ca7e403e1b/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/704c67fdceb66636af911e671db58a07833bd6ae70c9b2207a41baf55dab4bc7/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/14ddeb6bea0b19292e90c34685afd2bc38eede85d0e74ac075eec860908914c8/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/23ee3b59958e2e7669146074e6d6c617cf1c583307d95b06b078b9086a82ec9b/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/fd674dee122da583831750bbc859d3914152eb8b14a044c47cc22047cfb6dbdb/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/var/lib/containers/storage/overlay/f60084817b334ee1ad03a83101902e2e4eff798a91b4e211399d40dd0cec6073/merged 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:04:26.259 INFO:teuthology.orchestra.run.smithi073.stdout:/home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:26.260 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:26.260 DEBUG:teuthology.orchestra.run.smithi073:> ls /sys/fs/fuse/connections 2024-03-09T00:04:26.317 INFO:teuthology.orchestra.run.smithi073.stdout:119 2024-03-09T00:04:26.317 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [119] 2024-03-09T00:04:26.317 DEBUG:teuthology.orchestra.run.smithi073:> sudo stdin-killer -- python3 -c ' 2024-03-09T00:04:26.317 DEBUG:teuthology.orchestra.run.smithi073:> import glob 2024-03-09T00:04:26.317 DEBUG:teuthology.orchestra.run.smithi073:> import re 2024-03-09T00:04:26.317 DEBUG:teuthology.orchestra.run.smithi073:> import os 2024-03-09T00:04:26.317 DEBUG:teuthology.orchestra.run.smithi073:> import subprocess 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> def _find_admin_socket(client_name): 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> asok_path = "/var/run/ceph/ceph-client.admin.*.asok" 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> files = glob.glob(asok_path) 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> mountpoint = "/home/ubuntu/cephtest/mnt.admin" 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> # Given a non-glob path, it better be there 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> if "*" not in asok_path: 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> assert(len(files) == 1) 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> return files[0] 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> for f in files: 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2024-03-09T00:04:26.318 DEBUG:teuthology.orchestra.run.smithi073:> if os.path.exists("/proc/{0}".format(pid)): 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> with open("/proc/{0}/cmdline".format(pid), '"'"'r'"'"') as proc_f: 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> contents = proc_f.read() 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> if mountpoint in contents: 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> return f 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> print(_find_admin_socket("client.admin")) 2024-03-09T00:04:26.319 DEBUG:teuthology.orchestra.run.smithi073:> ' 2024-03-09T00:04:26.441 INFO:teuthology.orchestra.run.smithi073.stdout:/var/run/ceph/ceph-client.admin.76724.asok 2024-03-09T00:04:26.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26 stdin-killer INFO: command exited with status 0: exiting normally with same code! 2024-03-09T00:04:26.452 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.admin.76724.asok 2024-03-09T00:04:26.452 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:26.452 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.admin.76724.asok status 2024-03-09T00:04:26.590 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:04:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "metadata": { 2024-03-09T00:04:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_sha1": "7d63482aa89628d733e298984a762389b625c761", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_version": "ceph version 19.0.0-1965-g7d63482a (7d63482aa89628d733e298984a762389b625c761) squid (dev)", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "entity_id": "admin", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "hostname": "smithi073", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.admin", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "pid": "76724", 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "root": "/" 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "dentry_count": 0, 2024-03-09T00:04:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "dentry_pinned_count": 0, 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "id": 14784, 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "inst": { 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "name": { 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "client", 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "num": 14784 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": { 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v1", 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "192.168.0.1:0", 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 1524916144 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:04:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": { 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v1", 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "192.168.0.1:0", 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 1524916144 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "inst_str": "client.14784 192.168.0.1:0/1524916144", 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "addr_str": "192.168.0.1:0/1524916144", 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "inode_count": 1, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_epoch": 16, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_epoch": 89, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_epoch_barrier": 0, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "blocklisted": false, 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "fs_name": "cephfs" 2024-03-09T00:04:26.593 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:04:26.599 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:26.599 DEBUG:teuthology.orchestra.run.smithi073:> stat --file-system '--printf=%T 2024-03-09T00:04:26.599 DEBUG:teuthology.orchestra.run.smithi073:> ' -- /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:26.614 INFO:teuthology.orchestra.run.smithi073.stdout:fuseblk 2024-03-09T00:04:26.614 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:26.614 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:26.614 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:26.682 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:26.683 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.admin && exec bash -c 'setfattr -n ceph.dir.layout.pool -v cephfs_data_ec . && getfattr -n ceph.dir.layout .') 2024-03-09T00:04:26.754 INFO:teuthology.orchestra.run.smithi073.stdout:# file: . 2024-03-09T00:04:26.755 INFO:teuthology.orchestra.run.smithi073.stdout:ceph.dir.layout="stripe_unit=4194304 stripe_count=1 object_size=4194304 pool=cephfs_data_ec" 2024-03-09T00:04:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:26.755 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:04:26.755 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-09T00:04:26.811 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:26.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 -- 172.21.15.73:0/120390956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c101310 msgr2=0x7f124c101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:26.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 --2- 172.21.15.73:0/120390956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c101310 0x7f124c101730 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f1248009900 tx=0x7f124802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 -- 172.21.15.73:0/120390956 shutdown_connections 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 --2- 172.21.15.73:0/120390956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102ec0 0x7f124c109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 --2- 172.21.15.73:0/120390956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c102500 0x7f124c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 --2- 172.21.15.73:0/120390956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c101310 0x7f124c101730 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 -- 172.21.15.73:0/120390956 >> 172.21.15.73:0/120390956 conn(0x7f124c0fc8a0 msgr2=0x7f124c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 -- 172.21.15.73:0/120390956 shutdown_connections 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.957+0000 7f1254273640 1 -- 172.21.15.73:0/120390956 wait complete. 2024-03-09T00:04:26.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 Processor -- start 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 -- start start 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c101310 0x7f124c197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102500 0x7f124c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10cfa0 con 0x7f124c102ec0 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10ce20 con 0x7f124c101310 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f124c10d120 con 0x7f124c102500 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42374/0 (socket says 172.21.15.73:42374) 2024-03-09T00:04:26.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 -- 172.21.15.73:0/3310764479 learned_addr learned my addr 172.21.15.73:0/3310764479 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f125200f640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c101310 0x7f124c197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f125180e640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102500 0x7f124c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 -- 172.21.15.73:0/3310764479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102500 msgr2=0x7f124c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102500 0x7f124c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 -- 172.21.15.73:0/3310764479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c101310 msgr2=0x7f124c197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c101310 0x7f124c197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 -- 172.21.15.73:0/3310764479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f124c19f270 con 0x7f124c102ec0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1252810640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f124400cad0 tx=0x7f124400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124400ae20 con 0x7f124c102ec0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f124c19f560 con 0x7f124c102ec0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1244005020 con 0x7f124c102ec0 2024-03-09T00:04:26.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.958+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f124400f6b0 con 0x7f124c102ec0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.959+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f124c19fab0 con 0x7f124c102ec0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.959+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f124c108630 con 0x7f124c102ec0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.961+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12440051d0 con 0x7f124c102ec0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.961+0000 7f12437fe640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 0x7f12380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.961+0000 7f125200f640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 0x7f12380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.961+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f12440cf100 con 0x7f124c102ec0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.962+0000 7f125200f640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 0x7f12380a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1248000c00 tx=0x7f1248005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:26.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:26.966+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1244014030 con 0x7f124c102ec0 2024-03-09T00:04:27.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:26 smithi102 ceph-mon[36106]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-09T00:04:27.081 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:26 smithi179 ceph-mon[36637]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-09T00:04:27.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.142+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f124c061ef0 con 0x7f124c102ec0 2024-03-09T00:04:27.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.143+0000 7f12437fe640 1 -- 172.21.15.73:0/3310764479 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7f124409b910 con 0x7f124c102ec0 2024-03-09T00:04:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:27.140 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.145+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 msgr2=0x7f12380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 0x7f12380a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f1248000c00 tx=0x7f1248005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 msgr2=0x7f124c19ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f124400cad0 tx=0x7f124400cf90 comp rx=0 tx=0).stop 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 shutdown_connections 2024-03-09T00:04:27.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12380a2800 0x7f12380a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f124c102ec0 0x7f124c19ec00 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f124c102500 0x7f124c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 --2- 172.21.15.73:0/3310764479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f124c101310 0x7f124c197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 >> 172.21.15.73:0/3310764479 conn(0x7f124c0fc8a0 msgr2=0x7f124c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 shutdown_connections 2024-03-09T00:04:27.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.146+0000 7f1254273640 1 -- 172.21.15.73:0/3310764479 wait complete. 2024-03-09T00:04:27.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:26 smithi073 ceph-mon[32309]: pgmap v300: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 511 B/s wr, 1 op/s 2024-03-09T00:04:27.154 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:04:27.158 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:04:27.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- 172.21.15.73:0/3431805068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc104580 msgr2=0x7f75cc106970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- 172.21.15.73:0/3431805068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc104580 0x7f75cc106970 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f75b8009900 tx=0x7f75b802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:27.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- 172.21.15.73:0/3431805068 shutdown_connections 2024-03-09T00:04:27.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- 172.21.15.73:0/3431805068 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc106eb0 0x7f75cc109300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- 172.21.15.73:0/3431805068 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc104580 0x7f75cc106970 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- 172.21.15.73:0/3431805068 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc101c50 0x7f75cc104040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- 172.21.15.73:0/3431805068 >> 172.21.15.73:0/3431805068 conn(0x7f75cc0fb660 msgr2=0x7f75cc0fda80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- 172.21.15.73:0/3431805068 shutdown_connections 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- 172.21.15.73:0/3431805068 wait complete. 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 Processor -- start 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- start start 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 0x7f75cc1973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 0x7f75cc197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc10ce60 con 0x7f75cc106eb0 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc10cce0 con 0x7f75cc104580 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d1da2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75cc10cfe0 con 0x7f75cc101c50 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d15a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d15a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42398/0 (socket says 172.21.15.73:42398) 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.271+0000 7f75d15a1640 1 -- 172.21.15.73:0/1680638066 learned_addr learned my addr 172.21.15.73:0/1680638066 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d0da0640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 0x7f75cc1973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75cbfff640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 0x7f75cc197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 -- 172.21.15.73:0/1680638066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 msgr2=0x7f75cc1973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 0x7f75cc1973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 -- 172.21.15.73:0/1680638066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 msgr2=0x7f75cc197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 0x7f75cc197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 -- 172.21.15.73:0/1680638066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75cc19eff0 con 0x7f75cc106eb0 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d0da0640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 0x7f75cc1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75cbfff640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 0x7f75cc197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d15a1640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75c400cce0 tx=0x7f75c400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c400f450 con 0x7f75cc106eb0 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75cc19f2e0 con 0x7f75cc106eb0 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f75c400fad0 con 0x7f75cc106eb0 2024-03-09T00:04:27.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f75c4018c10 con 0x7f75cc106eb0 2024-03-09T00:04:27.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.272+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75cc19f830 con 0x7f75cc106eb0 2024-03-09T00:04:27.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.273+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75cc0fd150 con 0x7f75cc106eb0 2024-03-09T00:04:27.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.275+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f75c400f5f0 con 0x7f75cc106eb0 2024-03-09T00:04:27.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.275+0000 7f75c9ffb640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 0x7f75bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.275+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f75c40cf4c0 con 0x7f75cc106eb0 2024-03-09T00:04:27.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.275+0000 7f75d0da0640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 0x7f75bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.277+0000 7f75d0da0640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 0x7f75bc0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75c000abe0 tx=0x7f75c000a5c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.279+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f75c4014030 con 0x7f75cc106eb0 2024-03-09T00:04:27.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.461+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f75cc061ef0 con 0x7f75cc106eb0 2024-03-09T00:04:27.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.461+0000 7f75c9ffb640 1 -- 172.21.15.73:0/1680638066 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f75c409c3b0 con 0x7f75cc106eb0 2024-03-09T00:04:27.457 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:04:27.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 msgr2=0x7f75bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 0x7f75bc0a4cc0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f75c000abe0 tx=0x7f75c000a5c0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 msgr2=0x7f75cc19e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f75c400cce0 tx=0x7f75c400a620 comp rx=0 tx=0).stop 2024-03-09T00:04:27.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 shutdown_connections 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f75bc0a2800 0x7f75bc0a4cc0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f75cc106eb0 0x7f75cc19e980 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f75cc104580 0x7f75cc197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 --2- 172.21.15.73:0/1680638066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f75cc101c50 0x7f75cc1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 >> 172.21.15.73:0/1680638066 conn(0x7f75cc0fb660 msgr2=0x7f75cc102590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 shutdown_connections 2024-03-09T00:04:27.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.464+0000 7f75d1da2640 1 -- 172.21.15.73:0/1680638066 wait complete. 2024-03-09T00:04:27.470 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:04:27.580 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 -- 172.21.15.73:0/1909365525 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab14101da0 msgr2=0x7fab14108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.580 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 --2- 172.21.15.73:0/1909365525 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab14101da0 0x7fab14108630 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7fab0c009ca0 tx=0x7fab0c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 -- 172.21.15.73:0/1909365525 shutdown_connections 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 --2- 172.21.15.73:0/1909365525 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab14101da0 0x7fab14108630 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 --2- 172.21.15.73:0/1909365525 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab141013e0 0x7fab14101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 --2- 172.21.15.73:0/1909365525 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab141001f0 0x7fab14100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.583+0000 7fab1c085640 1 -- 172.21.15.73:0/1909365525 >> 172.21.15.73:0/1909365525 conn(0x7fab140fb780 msgr2=0x7fab140fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- 172.21.15.73:0/1909365525 shutdown_connections 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- 172.21.15.73:0/1909365525 wait complete. 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 Processor -- start 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- start start 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab141001f0 0x7fab141964b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14101da0 0x7fab1419da60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1410be80 con 0x7fab141013e0 2024-03-09T00:04:27.581 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1410bd00 con 0x7fab141001f0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1c085640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab1410c000 con 0x7fab14101da0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab19620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab19620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42422/0 (socket says 172.21.15.73:42422) 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab19620640 1 -- 172.21.15.73:0/3502817649 learned_addr learned my addr 172.21.15.73:0/3502817649 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.584+0000 7fab1a622640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14101da0 0x7fab1419da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 -- 172.21.15.73:0/3502817649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14101da0 msgr2=0x7fab1419da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14101da0 0x7fab1419da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 -- 172.21.15.73:0/3502817649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab141001f0 msgr2=0x7fab141964b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab141001f0 0x7fab141964b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 -- 172.21.15.73:0/3502817649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1419e0d0 con 0x7fab141013e0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab19620640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c890 tx=0x7fab1000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab10004220 con 0x7fab141013e0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab100043c0 con 0x7fab141013e0 2024-03-09T00:04:27.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab1419e3c0 con 0x7fab141013e0 2024-03-09T00:04:27.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab100049e0 con 0x7fab141013e0 2024-03-09T00:04:27.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.585+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab1419e910 con 0x7fab141013e0 2024-03-09T00:04:27.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.586+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab14107510 con 0x7fab141013e0 2024-03-09T00:04:27.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.587+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fab100079b0 con 0x7fab141013e0 2024-03-09T00:04:27.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.587+0000 7fab0affd640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 0x7fab000a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.588+0000 7fab19e21640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 0x7fab000a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.588+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fab100cee10 con 0x7fab141013e0 2024-03-09T00:04:27.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.588+0000 7fab19e21640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 0x7fab000a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab14101240 tx=0x7fab04009630 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.592+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fab10014030 con 0x7fab141013e0 2024-03-09T00:04:27.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.771+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fab1402ce70 con 0x7fab141013e0 2024-03-09T00:04:27.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.772+0000 7fab0affd640 1 -- 172.21.15.73:0/3502817649 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7fab1402ce70 con 0x7fab141013e0 2024-03-09T00:04:27.766 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 msgr2=0x7fab000a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 0x7fab000a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fab14101240 tx=0x7fab04009630 comp rx=0 tx=0).stop 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 msgr2=0x7fab141969f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fab1000c890 tx=0x7fab1000cd50 comp rx=0 tx=0).stop 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 shutdown_connections 2024-03-09T00:04:27.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fab000a2800 0x7fab000a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fab14101da0 0x7fab1419da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fab141013e0 0x7fab141969f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 --2- 172.21.15.73:0/3502817649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fab141001f0 0x7fab141964b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 >> 172.21.15.73:0/3502817649 conn(0x7fab140fb780 msgr2=0x7fab141046e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 shutdown_connections 2024-03-09T00:04:27.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.774+0000 7fab1c085640 1 -- 172.21.15.73:0/3502817649 wait complete. 2024-03-09T00:04:27.780 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:04:27.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 -- 172.21.15.73:0/814261975 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80101013e0 msgr2=0x7f8010101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 --2- 172.21.15.73:0/814261975 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80101013e0 0x7f8010101860 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f800c009ca0 tx=0x7f800c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 -- 172.21.15.73:0/814261975 shutdown_connections 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 --2- 172.21.15.73:0/814261975 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8010101da0 0x7f8010108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 --2- 172.21.15.73:0/814261975 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f80101013e0 0x7f8010101860 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 --2- 172.21.15.73:0/814261975 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f8010100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 -- 172.21.15.73:0/814261975 >> 172.21.15.73:0/814261975 conn(0x7f80100fb780 msgr2=0x7f80100fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 -- 172.21.15.73:0/814261975 shutdown_connections 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.893+0000 7f8017a85640 1 -- 172.21.15.73:0/814261975 wait complete. 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 Processor -- start 2024-03-09T00:04:27.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 -- start start 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f80101964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 0x7f8010196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801010be80 con 0x7f8010101da0 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801010bd00 con 0x7f80101001f0 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8017a85640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f801010c000 con 0x7f80101013e0 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42452/0 (socket says 172.21.15.73:42452) 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 -- 172.21.15.73:0/987474075 learned_addr learned my addr 172.21.15.73:0/987474075 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8015821640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f80101964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8015020640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 0x7f8010196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 -- 172.21.15.73:0/987474075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 msgr2=0x7f8010196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 0x7f8010196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 -- 172.21.15.73:0/987474075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 msgr2=0x7f80101964e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f80101964e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 -- 172.21.15.73:0/987474075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801019e100 con 0x7f8010101da0 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8015020640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 0x7f8010196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8015821640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f80101964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.894+0000 7f8016022640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f800400e970 tx=0x7f800400ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.895+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f800400ccc0 con 0x7f8010101da0 2024-03-09T00:04:27.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.895+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f800400ce60 con 0x7f8010101da0 2024-03-09T00:04:27.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.895+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f801019e3f0 con 0x7f8010101da0 2024-03-09T00:04:27.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.895+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8004010490 con 0x7f8010101da0 2024-03-09T00:04:27.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.895+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f801019e8c0 con 0x7f8010101da0 2024-03-09T00:04:27.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.896+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8010107530 con 0x7f8010101da0 2024-03-09T00:04:27.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.897+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8004005020 con 0x7f8010101da0 2024-03-09T00:04:27.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.898+0000 7f8002ffd640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 0x7f7ffc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:27.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.898+0000 7f8015821640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 0x7f7ffc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:27.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.898+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f80040cf1e0 con 0x7f8010101da0 2024-03-09T00:04:27.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.898+0000 7f8015821640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 0x7f7ffc0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f80080058c0 tx=0x7f8008005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:27.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:27.902+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f800409c0e0 con 0x7f8010101da0 2024-03-09T00:04:28.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.099+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8010061ef0 con 0x7f8010101da0 2024-03-09T00:04:28.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.099+0000 7f8002ffd640 1 -- 172.21.15.73:0/987474075 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f800409b9e0 con 0x7f8010101da0 2024-03-09T00:04:28.095 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 msgr2=0x7f7ffc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 0x7f7ffc0a4cc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f80080058c0 tx=0x7f8008005850 comp rx=0 tx=0).stop 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 msgr2=0x7f801019da90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f800400e970 tx=0x7f800400ed70 comp rx=0 tx=0).stop 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 shutdown_connections 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7ffc0a2800 0x7f7ffc0a4cc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8010101da0 0x7f801019da90 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f80101013e0 0x7f8010196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 --2- 172.21.15.73:0/987474075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f80101001f0 0x7f80101964e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 >> 172.21.15.73:0/987474075 conn(0x7f80100fb780 msgr2=0x7f8010104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 shutdown_connections 2024-03-09T00:04:28.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.102+0000 7f8017a85640 1 -- 172.21.15.73:0/987474075 wait complete. 2024-03-09T00:04:28.106 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3310764479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:28.106 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1680638066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.106 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3502817649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.107 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:04:28.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 -- 172.21.15.73:0/524491776 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841013e0 msgr2=0x7f9d84101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 --2- 172.21.15.73:0/524491776 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841013e0 0x7f9d84101860 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c009900 tx=0x7f9d6c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:28.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 -- 172.21.15.73:0/524491776 shutdown_connections 2024-03-09T00:04:28.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 --2- 172.21.15.73:0/524491776 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d84101da0 0x7f9d84108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 --2- 172.21.15.73:0/524491776 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841013e0 0x7f9d84101860 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 --2- 172.21.15.73:0/524491776 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d841001f0 0x7f9d84100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 -- 172.21.15.73:0/524491776 >> 172.21.15.73:0/524491776 conn(0x7f9d840fb780 msgr2=0x7f9d840fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 -- 172.21.15.73:0/524491776 shutdown_connections 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 -- 172.21.15.73:0/524491776 wait complete. 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.219+0000 7f9d89f62640 1 Processor -- start 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 -- start start 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 0x7f9d84197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 0x7f9d8419ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8410be80 con 0x7f9d841001f0 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8410bd00 con 0x7f9d841013e0 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d89f62640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d8410c000 con 0x7f9d84101da0 2024-03-09T00:04:28.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42468/0 (socket says 172.21.15.73:42468) 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 -- 172.21.15.73:0/748009050 learned_addr learned my addr 172.21.15.73:0/748009050 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d83fff640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 0x7f9d8419ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d82ffd640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 0x7f9d84197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 -- 172.21.15.73:0/748009050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 msgr2=0x7f9d8419ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 0x7f9d8419ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 -- 172.21.15.73:0/748009050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 msgr2=0x7f9d84197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 0x7f9d84197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 -- 172.21.15.73:0/748009050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8419f230 con 0x7f9d841001f0 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d82ffd640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 0x7f9d84197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d83fff640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 0x7f9d8419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.220+0000 7f9d837fe640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400eaa0 tx=0x7f9d7400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.221+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d74016070 con 0x7f9d841001f0 2024-03-09T00:04:28.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.221+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d8419f520 con 0x7f9d841001f0 2024-03-09T00:04:28.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.221+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d74003e50 con 0x7f9d841001f0 2024-03-09T00:04:28.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.221+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d740116d0 con 0x7f9d841001f0 2024-03-09T00:04:28.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.221+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d8419f9f0 con 0x7f9d841001f0 2024-03-09T00:04:28.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.222+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d8419fcb0 con 0x7f9d841001f0 2024-03-09T00:04:28.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.223+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f9d74019070 con 0x7f9d841001f0 2024-03-09T00:04:28.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.223+0000 7f9d80ff9640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 0x7f9d700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.223+0000 7f9d82ffd640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 0x7f9d700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.223+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f9d740cf1f0 con 0x7f9d841001f0 2024-03-09T00:04:28.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.224+0000 7f9d82ffd640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 0x7f9d700a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006010 tx=0x7f9d6c02c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.231+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9d7409c050 con 0x7f9d841001f0 2024-03-09T00:04:28.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:27 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3310764479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:28.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:27 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1680638066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:27 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3502817649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:27 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3310764479' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:27 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1680638066' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:27 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3502817649' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:28.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.411+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f9d8419fe80 con 0x7f9d841001f0 2024-03-09T00:04:28.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.412+0000 7f9d80ff9640 1 -- 172.21.15.73:0/748009050 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7f9d7409b950 con 0x7f9d841001f0 2024-03-09T00:04:28.407 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:04:28.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 msgr2=0x7f9d700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 0x7f9d700a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f9d6c006010 tx=0x7f9d6c02c8a0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 msgr2=0x7f9d84197610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9d7400eaa0 tx=0x7f9d7400ef60 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 shutdown_connections 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9d700a2800 0x7f9d700a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d84101da0 0x7f9d8419ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d841013e0 0x7f9d84197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 --2- 172.21.15.73:0/748009050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d841001f0 0x7f9d84197610 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 >> 172.21.15.73:0/748009050 conn(0x7f9d840fb780 msgr2=0x7f9d84068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 shutdown_connections 2024-03-09T00:04:28.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.414+0000 7f9d89f62640 1 -- 172.21.15.73:0/748009050 wait complete. 2024-03-09T00:04:28.420 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-09T00:04:28.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 -- 172.21.15.73:0/3766992787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc102f60 msgr2=0x7f6abc1097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/3766992787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc102f60 0x7f6abc1097f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f6ab4009cd0 tx=0x7f6ab402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 -- 172.21.15.73:0/3766992787 shutdown_connections 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/3766992787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc102f60 0x7f6abc1097f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/3766992787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc1025a0 0x7f6abc102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/3766992787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc0ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 -- 172.21.15.73:0/3766992787 >> 172.21.15.73:0/3766992787 conn(0x7f6abc0faf00 msgr2=0x7f6abc0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 -- 172.21.15.73:0/3766992787 shutdown_connections 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.528+0000 7f6ac1f90640 1 -- 172.21.15.73:0/3766992787 wait complete. 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 Processor -- start 2024-03-09T00:04:28.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 -- start start 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc1025a0 0x7f6abc197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc102f60 0x7f6abc19eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc10cfc0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc10ce40 con 0x7f6abc102f60 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6ac1f90640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6abc10d140 con 0x7f6abc1025a0 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6abb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34600/0 (socket says 172.21.15.73:34600) 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6abb7fe640 1 -- 172.21.15.73:0/2707300881 learned_addr learned my addr 172.21.15.73:0/2707300881 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.529+0000 7f6abaffd640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc1025a0 0x7f6abc197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abbfff640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc102f60 0x7f6abc19eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 -- 172.21.15.73:0/2707300881 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc1025a0 msgr2=0x7f6abc197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc1025a0 0x7f6abc197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 -- 172.21.15.73:0/2707300881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc102f60 msgr2=0x7f6abc19eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc102f60 0x7f6abc19eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 -- 172.21.15.73:0/2707300881 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6abc19f1e0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6abb7fe640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00d770 tx=0x7f6aac00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac017070 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6aac009c30 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6abc19f4d0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6aac003dd0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.530+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6abc19fa20 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.531+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6abc108720 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.532+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6aac003fb0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.532+0000 7f6ab8ff9640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 0x7f6aa40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.533+0000 7f6abaffd640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 0x7f6aa40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.533+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f6aac0ce6d0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.533+0000 7f6abaffd640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 0x7f6aa40a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6abc198ad0 tx=0x7f6ab000a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.537+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6aac09b530 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.726+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f6abc02d030 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.726+0000 7f6ab8ff9640 1 -- 172.21.15.73:0/2707300881 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+589 (secure 0 0 0) 0x7f6aac009de0 con 0x7f6abc0ff5d0 2024-03-09T00:04:28.722 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:28.722 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/3726094398 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/1410011990 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/545419019 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/4143923877 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/161900832 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:28.723 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/383471347 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:listed 10 entries 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 msgr2=0x7f6aa40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 0x7f6aa40a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f6abc198ad0 tx=0x7f6ab000a4c0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 msgr2=0x7f6abc1975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f6aac00d770 tx=0x7f6aac00dc30 comp rx=0 tx=0).stop 2024-03-09T00:04:28.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 shutdown_connections 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6aa40a2800 0x7f6aa40a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6abc102f60 0x7f6abc19eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6abc1025a0 0x7f6abc197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 --2- 172.21.15.73:0/2707300881 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6abc0ff5d0 0x7f6abc1975c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 >> 172.21.15.73:0/2707300881 conn(0x7f6abc0faf00 msgr2=0x7f6abc1058f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.728+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 shutdown_connections 2024-03-09T00:04:28.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.729+0000 7f6ac1f90640 1 -- 172.21.15.73:0/2707300881 wait complete. 2024-03-09T00:04:28.734 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:04:28.735 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-09T00:04:28.752 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 -- 172.21.15.73:0/2065302045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0101400 msgr2=0x7fabf0101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/2065302045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0101400 0x7fabf0101880 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fabe4009900 tx=0x7fabe402bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 -- 172.21.15.73:0/2065302045 shutdown_connections 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/2065302045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101dc0 0x7fabf0108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/2065302045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0101400 0x7fabf0101880 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/2065302045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0100210 0x7fabf0100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.898+0000 7fabf6f0a640 1 -- 172.21.15.73:0/2065302045 >> 172.21.15.73:0/2065302045 conn(0x7fabf00fb780 msgr2=0x7fabf00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- 172.21.15.73:0/2065302045 shutdown_connections 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- 172.21.15.73:0/2065302045 wait complete. 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 Processor -- start 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- start start 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101400 0x7fabf019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 0x7fabf019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf010bea0 con 0x7fabf0100210 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf010bd20 con 0x7fabf0101400 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf6f0a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabf010c020 con 0x7fabf0101dc0 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf4ca6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf4ca6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34604/0 (socket says 172.21.15.73:34604) 2024-03-09T00:04:28.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf4ca6640 1 -- 172.21.15.73:0/1025664562 learned_addr learned my addr 172.21.15.73:0/1025664562 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabeffff640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101400 0x7fabf019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.899+0000 7fabf54a7640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 0x7fabf019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 -- 172.21.15.73:0/1025664562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 msgr2=0x7fabf019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 0x7fabf019e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 -- 172.21.15.73:0/1025664562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101400 msgr2=0x7fabf019c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101400 0x7fabf019c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 -- 172.21.15.73:0/1025664562 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabf019f230 con 0x7fabf0100210 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf54a7640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 0x7fabf019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf4ca6640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fabe000d950 tx=0x7fabe000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe0017020 con 0x7fabf0100210 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabf019f520 con 0x7fabf0100210 2024-03-09T00:04:28.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabe0009c30 con 0x7fabf0100210 2024-03-09T00:04:28.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabe000f740 con 0x7fabf0100210 2024-03-09T00:04:28.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.900+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabf019fa90 con 0x7fabf0100210 2024-03-09T00:04:28.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.901+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabf0107530 con 0x7fabf0100210 2024-03-09T00:04:28.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.902+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fabe001a020 con 0x7fabf0100210 2024-03-09T00:04:28.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.903+0000 7fabedffb640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 0x7fabd80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:28.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.903+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fabe00cea10 con 0x7fabf0100210 2024-03-09T00:04:28.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.903+0000 7fabeffff640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 0x7fabd80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:28.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.903+0000 7fabeffff640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 0x7fabd80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabe402c7b0 tx=0x7fabe4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:28.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:28.906+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fabe0021070 con 0x7fabf0100210 2024-03-09T00:04:29.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.084+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fabf002d0d0 con 0x7fabf0100210 2024-03-09T00:04:29.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.086+0000 7fabedffb640 1 -- 172.21.15.73:0/1025664562 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7fabe0009de0 con 0x7fabf0100210 2024-03-09T00:04:29.081 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:29.081 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 msgr2=0x7fabd80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 0x7fabd80a4cc0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fabe402c7b0 tx=0x7fabe4005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 msgr2=0x7fabf0068fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fabe000d950 tx=0x7fabe000de10 comp rx=0 tx=0).stop 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 shutdown_connections 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabd80a2800 0x7fabd80a4cc0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabf0101dc0 0x7fabf019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabf0101400 0x7fabf019c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 --2- 172.21.15.73:0/1025664562 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabf0100210 0x7fabf0068fe0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 >> 172.21.15.73:0/1025664562 conn(0x7fabf00fb780 msgr2=0x7fabf0104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 shutdown_connections 2024-03-09T00:04:29.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.088+0000 7fabf6f0a640 1 -- 172.21.15.73:0/1025664562 wait complete. 2024-03-09T00:04:29.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:28 smithi073 ceph-mon[32309]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 1 op/s 2024-03-09T00:04:29.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:28 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/987474075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:28 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/748009050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.093 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:28 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2707300881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:29.094 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:04:29.095 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 -- 172.21.15.73:0/3479641947 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 msgr2=0x7f4f70100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 --2- 172.21.15.73:0/3479641947 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70100610 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f4f60009900 tx=0x7f4f6002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 -- 172.21.15.73:0/3479641947 shutdown_connections 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 --2- 172.21.15.73:0/3479641947 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 0x7f4f70108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 --2- 172.21.15.73:0/3479641947 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 0x7f4f70101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 --2- 172.21.15.73:0/3479641947 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70100610 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 -- 172.21.15.73:0/3479641947 >> 172.21.15.73:0/3479641947 conn(0x7f4f700fb780 msgr2=0x7f4f700fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 -- 172.21.15.73:0/3479641947 shutdown_connections 2024-03-09T00:04:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.205+0000 7f4f7703d640 1 -- 172.21.15.73:0/3479641947 wait complete. 2024-03-09T00:04:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 Processor -- start 2024-03-09T00:04:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 -- start start 2024-03-09T00:04:29.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 0x7f4f70196b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 0x7f4f7019dbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7010bec0 con 0x7f4f701001f0 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7010bd40 con 0x7f4f70101da0 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f7703d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f7010c040 con 0x7f4f701013e0 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f74dd9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f74dd9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34622/0 (socket says 172.21.15.73:34622) 2024-03-09T00:04:29.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f74dd9640 1 -- 172.21.15.73:0/996145608 learned_addr learned my addr 172.21.15.73:0/996145608 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.206+0000 7f4f6ffff640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 0x7f4f70196b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f755da640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 0x7f4f7019dbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 -- 172.21.15.73:0/996145608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 msgr2=0x7f4f70196b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 0x7f4f70196b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 -- 172.21.15.73:0/996145608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 msgr2=0x7f4f7019dbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 0x7f4f7019dbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 -- 172.21.15.73:0/996145608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f7019e220 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f74dd9640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4f60000c00 tx=0x7f4f60004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f60041070 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4f600054e0 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f7019e4b0 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4f600346a0 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.207+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f7019e9a0 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.208+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f70107630 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.209+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4f60044070 con 0x7f4f701001f0 2024-03-09T00:04:29.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.210+0000 7f4f6dffb640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 0x7f4f580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.210+0000 7f4f6ffff640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 0x7f4f580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.210+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f4f600f0a60 con 0x7f4f701001f0 2024-03-09T00:04:29.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.210+0000 7f4f6ffff640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 0x7f4f580a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f70197b10 tx=0x7f4f64009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.214+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4f600bd8c0 con 0x7f4f701001f0 2024-03-09T00:04:29.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:28 smithi102 ceph-mon[36106]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 1 op/s 2024-03-09T00:04:29.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:28 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/987474075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:28 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/748009050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:28 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2707300881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:29.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:28 smithi179 ceph-mon[36637]: pgmap v301: 129 pgs: 129 active+clean; 579 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 410 B/s wr, 1 op/s 2024-03-09T00:04:29.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:28 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/987474075' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:28 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/748009050' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:29.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:28 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2707300881' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:29.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.389+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4f70061ef0 con 0x7f4f701001f0 2024-03-09T00:04:29.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.389+0000 7f4f6dffb640 1 -- 172.21.15.73:0/996145608 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v89) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4f60003e20 con 0x7f4f701001f0 2024-03-09T00:04:29.385 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 msgr2=0x7f4f580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 0x7f4f580a4cc0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f4f70197b10 tx=0x7f4f64009110 comp rx=0 tx=0).stop 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 msgr2=0x7f4f70196600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f4f60000c00 tx=0x7f4f60004040 comp rx=0 tx=0).stop 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 shutdown_connections 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4f580a2800 0x7f4f580a4cc0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4f70101da0 0x7f4f7019dbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4f701013e0 0x7f4f70196b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 --2- 172.21.15.73:0/996145608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4f701001f0 0x7f4f70196600 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 >> 172.21.15.73:0/996145608 conn(0x7f4f700fb780 msgr2=0x7f4f70104800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 shutdown_connections 2024-03-09T00:04:29.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.392+0000 7f4f7703d640 1 -- 172.21.15.73:0/996145608 wait complete. 2024-03-09T00:04:29.398 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:04:29.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 -- 172.21.15.73:0/1114926735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740101d90 msgr2=0x7f5740104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 --2- 172.21.15.73:0/1114926735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740101d90 0x7f5740104180 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f5730009900 tx=0x7f573002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 -- 172.21.15.73:0/1114926735 shutdown_connections 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 --2- 172.21.15.73:0/1114926735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740106ff0 0x7f5740109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 --2- 172.21.15.73:0/1114926735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 --2- 172.21.15.73:0/1114926735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740101d90 0x7f5740104180 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 -- 172.21.15.73:0/1114926735 >> 172.21.15.73:0/1114926735 conn(0x7f57400fb7a0 msgr2=0x7f57400fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 -- 172.21.15.73:0/1114926735 shutdown_connections 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.511+0000 7f57469b3640 1 -- 172.21.15.73:0/1114926735 wait complete. 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 Processor -- start 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 -- start start 2024-03-09T00:04:29.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740101d90 0x7f5740197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010cfa0 con 0x7f5740106ff0 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010ce20 con 0x7f5740101d90 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f57469b3640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f574010d120 con 0x7f57401046c0 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34636/0 (socket says 172.21.15.73:34636) 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 -- 172.21.15.73:0/145870210 learned_addr learned my addr 172.21.15.73:0/145870210 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:29.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f573f7fe640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 -- 172.21.15.73:0/145870210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 msgr2=0x7f5740197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f573ffff640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740101d90 0x7f5740197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 -- 172.21.15.73:0/145870210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740101d90 msgr2=0x7f5740197640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740101d90 0x7f5740197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f5744f50640 1 -- 172.21.15.73:0/145870210 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f574019f260 con 0x7f5740106ff0 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.512+0000 7f573f7fe640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f5744f50640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f573800eaa0 tx=0x7f573800ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f573800cdb0 con 0x7f5740106ff0 2024-03-09T00:04:29.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f574019f550 con 0x7f5740106ff0 2024-03-09T00:04:29.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5738005020 con 0x7f5740106ff0 2024-03-09T00:04:29.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f57380116b0 con 0x7f5740106ff0 2024-03-09T00:04:29.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.513+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f574019faa0 con 0x7f5740106ff0 2024-03-09T00:04:29.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.514+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57400fd290 con 0x7f5740106ff0 2024-03-09T00:04:29.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.515+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5738005560 con 0x7f5740106ff0 2024-03-09T00:04:29.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.515+0000 7f573d7fa640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.515+0000 7f573ffff640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.515+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f57380cefb0 con 0x7f5740106ff0 2024-03-09T00:04:29.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.516+0000 7f573ffff640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 0x7f57280a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5730005c20 tx=0x7f5730005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.520+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5738016030 con 0x7f5740106ff0 2024-03-09T00:04:29.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.691+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5740061ef0 con 0x7f5740106ff0 2024-03-09T00:04:29.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.692+0000 7f573d7fa640 1 -- 172.21.15.73:0/145870210 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v89) v1 ==== 104+0+11 (secure 0 0 0) 0x7f573809be10 con 0x7f5740106ff0 2024-03-09T00:04:29.687 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:04:29.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 msgr2=0x7f57280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 0x7f57280a4cc0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5730005c20 tx=0x7f5730005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 msgr2=0x7f574019ebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f573800eaa0 tx=0x7f573800ef60 comp rx=0 tx=0).stop 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 shutdown_connections 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f57280a2800 0x7f57280a4cc0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5740106ff0 0x7f574019ebf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f57401046c0 0x7f5740197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 --2- 172.21.15.73:0/145870210 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5740101d90 0x7f5740197640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 >> 172.21.15.73:0/145870210 conn(0x7f57400fb7a0 msgr2=0x7f57401027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 shutdown_connections 2024-03-09T00:04:29.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.694+0000 7f57469b3640 1 -- 172.21.15.73:0/145870210 wait complete. 2024-03-09T00:04:29.700 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/311662860 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d58102f60 msgr2=0x7f3d581097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/311662860 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d58102f60 0x7f3d581097f0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f3d50009cd0 tx=0x7f3d5002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/311662860 shutdown_connections 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/311662860 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d58102f60 0x7f3d581097f0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/311662860 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d581025a0 0x7f3d58102a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/311662860 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d580ff5d0 0x7f3d580ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/311662860 >> 172.21.15.73:0/311662860 conn(0x7f3d580faf00 msgr2=0x7f3d580fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/311662860 shutdown_connections 2024-03-09T00:04:29.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/311662860 wait complete. 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.810+0000 7f3d5d1b3640 1 Processor -- start 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 -- start start 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d581025a0 0x7f3d58197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d58102f60 0x7f3d5819ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5810cfc0 con 0x7f3d580ff5d0 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5810ce40 con 0x7f3d581025a0 2024-03-09T00:04:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d5d1b3640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d5810d140 con 0x7f3d58102f60 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34664/0 (socket says 172.21.15.73:34664) 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 -- 172.21.15.73:0/357691236 learned_addr learned my addr 172.21.15.73:0/357691236 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 -- 172.21.15.73:0/357691236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d58102f60 msgr2=0x7f3d5819ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d58102f60 0x7f3d5819ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 -- 172.21.15.73:0/357691236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d581025a0 msgr2=0x7f3d58197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d581025a0 0x7f3d58197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 -- 172.21.15.73:0/357691236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5819f220 con 0x7f3d580ff5d0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d56d9d640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d770 tx=0x7f3d4800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.811+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48017070 con 0x7f3d580ff5d0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.812+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d5819f510 con 0x7f3d580ff5d0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.812+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d48009c30 con 0x7f3d580ff5d0 2024-03-09T00:04:29.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.812+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d48003dd0 con 0x7f3d580ff5d0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.812+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d5819fab0 con 0x7f3d580ff5d0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.813+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d58108670 con 0x7f3d580ff5d0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.814+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3d48003f70 con 0x7f3d580ff5d0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.814+0000 7f3d3ffff640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 0x7f3d400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.814+0000 7f3d5659c640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 0x7f3d400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.814+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3d480cead0 con 0x7f3d580ff5d0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.815+0000 7f3d5659c640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 0x7f3d400a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c005fd0 tx=0x7f3d4c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.818+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3d48009de0 con 0x7f3d580ff5d0 2024-03-09T00:04:29.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.997+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d5802d060 con 0x7f3d580ff5d0 2024-03-09T00:04:29.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.997+0000 7f3d3ffff640 1 -- 172.21.15.73:0/357691236 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v89) v1 ==== 100+0+11 (secure 0 0 0) 0x7f3d4809b930 con 0x7f3d580ff5d0 2024-03-09T00:04:29.992 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 msgr2=0x7f3d400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 0x7f3d400a4cc0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f3d4c005fd0 tx=0x7f3d4c005850 comp rx=0 tx=0).stop 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 msgr2=0x7f3d58197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f3d4800d770 tx=0x7f3d4800dc30 comp rx=0 tx=0).stop 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 shutdown_connections 2024-03-09T00:04:29.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d400a2800 0x7f3d400a4cc0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d58102f60 0x7f3d5819ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d581025a0 0x7f3d58197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 --2- 172.21.15.73:0/357691236 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d580ff5d0 0x7f3d58197600 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 >> 172.21.15.73:0/357691236 conn(0x7f3d580faf00 msgr2=0x7f3d58105840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 shutdown_connections 2024-03-09T00:04:29.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:29.999+0000 7f3d5d1b3640 1 -- 172.21.15.73:0/357691236 wait complete. 2024-03-09T00:04:30.005 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:04:30.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 -- 172.21.15.73:0/2042131549 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 msgr2=0x7ffa78108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 --2- 172.21.15.73:0/2042131549 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa78108640 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7ffa70009ca0 tx=0x7ffa7002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 -- 172.21.15.73:0/2042131549 shutdown_connections 2024-03-09T00:04:30.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 --2- 172.21.15.73:0/2042131549 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa78108640 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 --2- 172.21.15.73:0/2042131549 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 0x7ffa78101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 --2- 172.21.15.73:0/2042131549 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 0x7ffa78100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 -- 172.21.15.73:0/2042131549 >> 172.21.15.73:0/2042131549 conn(0x7ffa780fb790 msgr2=0x7ffa780fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 -- 172.21.15.73:0/2042131549 shutdown_connections 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.115+0000 7ffa7db97640 1 -- 172.21.15.73:0/2042131549 wait complete. 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 Processor -- start 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 -- start start 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 0x7ffa7806a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 0x7ffa7819c360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810be90 con 0x7ffa78101db0 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810bd10 con 0x7ffa781013f0 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa7db97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa7810c010 con 0x7ffa78100200 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34680/0 (socket says 172.21.15.73:34680) 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 -- 172.21.15.73:0/1053843379 learned_addr learned my addr 172.21.15.73:0/1053843379 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa76ffd640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 0x7ffa7819c360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa777fe640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 0x7ffa7806a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 -- 172.21.15.73:0/1053843379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 msgr2=0x7ffa7806a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 0x7ffa7806a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 -- 172.21.15.73:0/1053843379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 msgr2=0x7ffa7819c360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.116+0000 7ffa77fff640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 0x7ffa7819c360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa77fff640 1 -- 172.21.15.73:0/1053843379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa7819f290 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa77fff640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002c700 tx=0x7ffa70032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa70039040 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa7819f4c0 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffa70044070 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffa700042a0 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.117+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa7819f9b0 con 0x7ffa78101db0 2024-03-09T00:04:30.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.118+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa78107520 con 0x7ffa78101db0 2024-03-09T00:04:30.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.119+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffa70003970 con 0x7ffa78101db0 2024-03-09T00:04:30.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.119+0000 7ffa74ff9640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 0x7ffa600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.119+0000 7ffa777fe640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 0x7ffa600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.119+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ffa700f1290 con 0x7ffa78101db0 2024-03-09T00:04:30.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.120+0000 7ffa777fe640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 0x7ffa600a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffa680058c0 tx=0x7ffa68005c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:30.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.124+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffa700333a0 con 0x7ffa78101db0 2024-03-09T00:04:30.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1025664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:30.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/996145608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/145870210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.297+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7ffa7802d070 con 0x7ffa78101db0 2024-03-09T00:04:30.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.297+0000 7ffa74ff9640 1 -- 172.21.15.73:0/1053843379 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v89) v1 ==== 103+0+11 (secure 0 0 0) 0x7ffa700be1a0 con 0x7ffa78101db0 2024-03-09T00:04:30.293 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 msgr2=0x7ffa600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 0x7ffa600a4cc0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7ffa680058c0 tx=0x7ffa68005c80 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 msgr2=0x7ffa7819e980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7ffa7002c700 tx=0x7ffa70032f20 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 shutdown_connections 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffa600a2800 0x7ffa600a4cc0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffa78101db0 0x7ffa7819e980 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffa781013f0 0x7ffa7819c360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 --2- 172.21.15.73:0/1053843379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffa78100200 0x7ffa7806a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 >> 172.21.15.73:0/1053843379 conn(0x7ffa780fb790 msgr2=0x7ffa781046f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.299+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 shutdown_connections 2024-03-09T00:04:30.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.300+0000 7ffa7db97640 1 -- 172.21.15.73:0/1053843379 wait complete. 2024-03-09T00:04:30.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1025664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:30.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/996145608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.304 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/145870210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.306 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-09T00:04:30.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1025664562' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:30.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/996145608' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/145870210' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:04:30.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- 172.21.15.73:0/925054647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb20102910 msgr2=0x7fbb20102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/925054647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb20102910 0x7fbb20102d30 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fbb14009900 tx=0x7fbb1402bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- 172.21.15.73:0/925054647 shutdown_connections 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/925054647 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb200ffd40 0x7fbb201098f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/925054647 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 0x7fbb200ff800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/925054647 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb20102910 0x7fbb20102d30 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- 172.21.15.73:0/925054647 >> 172.21.15.73:0/925054647 conn(0x7fbb200faf20 msgr2=0x7fbb200fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- 172.21.15.73:0/925054647 shutdown_connections 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- 172.21.15.73:0/925054647 wait complete. 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 Processor -- start 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.414+0000 7fbb26ec0640 1 -- start start 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 0x7fbb20197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 0x7fbb2019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2010d040 con 0x7fbb200ffd40 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2010cec0 con 0x7fbb20102910 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb2010d1c0 con 0x7fbb200ff380 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34696/0 (socket says 172.21.15.73:34696) 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 -- 172.21.15.73:0/655434507 learned_addr learned my addr 172.21.15.73:0/655434507 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb2545d640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 0x7fbb2019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb24c5c640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 0x7fbb20197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 -- 172.21.15.73:0/655434507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 msgr2=0x7fbb20197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 0x7fbb20197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 -- 172.21.15.73:0/655434507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 msgr2=0x7fbb2019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 0x7fbb2019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 -- 172.21.15.73:0/655434507 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb2019f2b0 con 0x7fbb200ffd40 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb2545d640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 0x7fbb2019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:30.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1ffff640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c970 tx=0x7fbb0c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c007bb0 con 0x7fbb200ffd40 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb2019f5a0 con 0x7fbb200ffd40 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbb0c007d50 con 0x7fbb200ffd40 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.415+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbb0c00f450 con 0x7fbb200ffd40 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.416+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb2019faf0 con 0x7fbb200ffd40 2024-03-09T00:04:30.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.417+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb20108850 con 0x7fbb200ffd40 2024-03-09T00:04:30.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.418+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbb0c005020 con 0x7fbb200ffd40 2024-03-09T00:04:30.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.418+0000 7fbb1dffb640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 0x7fbb100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:30.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.418+0000 7fbb24c5c640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 0x7fbb100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:30.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.418+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fbb0c0ceb20 con 0x7fbb200ffd40 2024-03-09T00:04:30.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.419+0000 7fbb24c5c640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 0x7fbb100a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb140096c0 tx=0x7fbb1402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:30.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.422+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbb0c014030 con 0x7fbb200ffd40 2024-03-09T00:04:30.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.610+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7fbb20061ef0 con 0x7fbb200ffd40 2024-03-09T00:04:30.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.611+0000 7fbb1dffb640 1 -- 172.21.15.73:0/655434507 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 10 entries v89) v1 ==== 81+0+589 (secure 0 0 0) 0x7fbb0c09b980 con 0x7fbb200ffd40 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/3726094398 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/1410011990 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/545419019 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:30.607 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/4143923877 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:30.608 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/161900832 2024-03-09T23:57:03.831065+0000 2024-03-09T00:04:30.608 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/383471347 2024-03-09T23:56:36.296712+0000 2024-03-09T00:04:30.609 INFO:teuthology.orchestra.run.smithi073.stderr:listed 10 entries 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 msgr2=0x7fbb100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 0x7fbb100a4cc0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fbb140096c0 tx=0x7fbb1402df90 comp rx=0 tx=0).stop 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 msgr2=0x7fbb20197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7fbb0c00c970 tx=0x7fbb0c00ce30 comp rx=0 tx=0).stop 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 shutdown_connections 2024-03-09T00:04:30.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbb100a2800 0x7fbb100a4cc0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbb20102910 0x7fbb2019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbb200ffd40 0x7fbb20197bd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 --2- 172.21.15.73:0/655434507 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbb200ff380 0x7fbb20197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 >> 172.21.15.73:0/655434507 conn(0x7fbb200faf20 msgr2=0x7fbb20105a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 shutdown_connections 2024-03-09T00:04:30.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:30.613+0000 7fbb26ec0640 1 -- 172.21.15.73:0/655434507 wait complete. 2024-03-09T00:04:30.619 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi073.front.sepia.ceph.com... 2024-03-09T00:04:30.619 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:30.619 DEBUG:teuthology.orchestra.run.smithi073:> sudo fusermount -u /home/ubuntu/cephtest/mnt.admin 2024-03-09T00:04:30.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.admin.smithi073.stderr:ceph-fuse[76724]: fuse finished with error 0 and tester_r 0 2024-03-09T00:04:30.646 INFO:teuthology.orchestra.run:waiting for 300 2024-03-09T00:04:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:30 smithi102 ceph-mon[36106]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 652 B/s wr, 1 op/s 2024-03-09T00:04:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/357691236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1053843379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/655434507' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:31.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:30 smithi179 ceph-mon[36637]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 652 B/s wr, 1 op/s 2024-03-09T00:04:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/357691236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1053843379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/655434507' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:30 smithi073 ceph-mon[32309]: pgmap v302: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 652 B/s wr, 1 op/s 2024-03-09T00:04:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/357691236' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1053843379' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:04:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/655434507' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:04:33.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:32 smithi102 ceph-mon[36106]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:33.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:32 smithi179 ceph-mon[36637]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:33.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:32 smithi073 ceph-mon[32309]: pgmap v303: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:35.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:34 smithi102 ceph-mon[36106]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:35.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:34 smithi179 ceph-mon[36637]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:35.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:34 smithi073 ceph-mon[32309]: pgmap v304: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 255 B/s wr, 0 op/s 2024-03-09T00:04:36.748 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:04:36.749 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:04:36.749 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.admin) 2024-03-09T00:04:36.781 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:36.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 -- 172.21.15.73:0/2929501353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8100290 msgr2=0x7fbaa81006b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:36.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2929501353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8100290 0x7fbaa81006b0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fba9c009900 tx=0x7fba9c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 -- 172.21.15.73:0/2929501353 shutdown_connections 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2929501353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8101e40 0x7fbaa81086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2929501353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101480 0x7fbaa8101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2929501353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8100290 0x7fbaa81006b0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 -- 172.21.15.73:0/2929501353 >> 172.21.15.73:0/2929501353 conn(0x7fbaa80fb800 msgr2=0x7fbaa80fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 -- 172.21.15.73:0/2929501353 shutdown_connections 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.941+0000 7fbaaca20640 1 -- 172.21.15.73:0/2929501353 wait complete. 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 Processor -- start 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 -- start start 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8100290 0x7fbaa8196580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101e40 0x7fbaa819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:36.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa810be80 con 0x7fbaa8101480 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa810bd00 con 0x7fbaa8100290 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaaca20640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaa810c000 con 0x7fbaa8101e40 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34720/0 (socket says 172.21.15.73:34720) 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 -- 172.21.15.73:0/2514099570 learned_addr learned my addr 172.21.15.73:0/2514099570 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa659c640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8100290 0x7fbaa8196580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa6d9d640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101e40 0x7fbaa819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 -- 172.21.15.73:0/2514099570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101e40 msgr2=0x7fbaa819db30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101e40 0x7fbaa819db30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 -- 172.21.15.73:0/2514099570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8100290 msgr2=0x7fbaa8196580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8100290 0x7fbaa8196580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 -- 172.21.15.73:0/2514099570 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaa819e1a0 con 0x7fbaa8101480 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.942+0000 7fbaa5d9b640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fba9800d950 tx=0x7fba9800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:36.940 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.943+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba98017030 con 0x7fbaa8101480 2024-03-09T00:04:36.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.943+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fba98004020 con 0x7fbaa8101480 2024-03-09T00:04:36.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.943+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaa819e490 con 0x7fbaa8101480 2024-03-09T00:04:36.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.943+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fba980047a0 con 0x7fbaa8101480 2024-03-09T00:04:36.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.943+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaa819e9e0 con 0x7fbaa8101480 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.944+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaa8107600 con 0x7fbaa8101480 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.945+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fba980041d0 con 0x7fbaa8101480 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.946+0000 7fba977fe640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 0x7fba900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.946+0000 7fbaa659c640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 0x7fba900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.946+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fba980ce7a0 con 0x7fbaa8101480 2024-03-09T00:04:36.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.946+0000 7fbaa659c640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 0x7fba900a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba9c02c7b0 tx=0x7fba9c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:36.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:36.950+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fba9809b600 con 0x7fbaa8101480 2024-03-09T00:04:37.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.126+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaa8061ef0 con 0x7fbaa8101480 2024-03-09T00:04:37.123 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.128+0000 7fba977fe640 1 -- 172.21.15.73:0/2514099570 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7fba9809af00 con 0x7fbaa8101480 2024-03-09T00:04:37.123 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:37.123 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 msgr2=0x7fba900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 0x7fba900a4cc0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fba9c02c7b0 tx=0x7fba9c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 msgr2=0x7fbaa8196ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fba9800d950 tx=0x7fba9800de10 comp rx=0 tx=0).stop 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 shutdown_connections 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fba900a2800 0x7fba900a4cc0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaa8101e40 0x7fbaa819db30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaa8101480 0x7fbaa8196ac0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 --2- 172.21.15.73:0/2514099570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaa8100290 0x7fbaa8196580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 >> 172.21.15.73:0/2514099570 conn(0x7fbaa80fb800 msgr2=0x7fbaa81047d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:37.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 shutdown_connections 2024-03-09T00:04:37.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.130+0000 7fbaaca20640 1 -- 172.21.15.73:0/2514099570 wait complete. 2024-03-09T00:04:37.135 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:36 smithi073 ceph-mon[32309]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:37.136 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:37.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 -- 172.21.15.73:0/165399027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c101400 msgr2=0x7f3f0c101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 --2- 172.21.15.73:0/165399027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c101400 0x7f3f0c101880 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f3efc009900 tx=0x7f3efc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 -- 172.21.15.73:0/165399027 shutdown_connections 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 --2- 172.21.15.73:0/165399027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101dc0 0x7f3f0c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 --2- 172.21.15.73:0/165399027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c101400 0x7f3f0c101880 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 --2- 172.21.15.73:0/165399027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c100210 0x7f3f0c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 -- 172.21.15.73:0/165399027 >> 172.21.15.73:0/165399027 conn(0x7f3f0c0fb780 msgr2=0x7f3f0c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 -- 172.21.15.73:0/165399027 shutdown_connections 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.244+0000 7f3f13390640 1 -- 172.21.15.73:0/165399027 wait complete. 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 Processor -- start 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 -- start start 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101400 0x7f3f0c197b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c101dc0 0x7f3f0c19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c10bea0 con 0x7f3f0c100210 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c10bd20 con 0x7f3f0c101400 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f13390640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f0c10c020 con 0x7f3f0c101dc0 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34734/0 (socket says 172.21.15.73:34734) 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 -- 172.21.15.73:0/2859320112 learned_addr learned my addr 172.21.15.73:0/2859320112 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1092b640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101400 0x7f3f0c197b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1192d640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c101dc0 0x7f3f0c19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 -- 172.21.15.73:0/2859320112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c101dc0 msgr2=0x7f3f0c19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c101dc0 0x7f3f0c19eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 -- 172.21.15.73:0/2859320112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101400 msgr2=0x7f3f0c197b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101400 0x7f3f0c197b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 -- 172.21.15.73:0/2859320112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f0c19f200 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f1112c640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800d950 tx=0x7f3f0800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.245+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f08017070 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.246+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f08009c30 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.246+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f0c19f4f0 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.246+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f0800f600 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.246+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f0c19f9c0 con 0x7f3f0c100210 2024-03-09T00:04:37.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.246+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f0c19fc20 con 0x7f3f0c100210 2024-03-09T00:04:37.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.247+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3f0801a070 con 0x7f3f0c100210 2024-03-09T00:04:37.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.248+0000 7f3f027fc640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 0x7f3ef80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.248+0000 7f3f1092b640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 0x7f3ef80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.248+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f3f080ce7b0 con 0x7f3f0c100210 2024-03-09T00:04:37.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.248+0000 7f3f1092b640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 0x7f3ef80a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006010 tx=0x7f3efc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:37.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.252+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3f08009de0 con 0x7f3f0c100210 2024-03-09T00:04:37.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:36 smithi102 ceph-mon[36106]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:37.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:36 smithi179 ceph-mon[36637]: pgmap v305: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:37.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.431+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3f0c19ff10 con 0x7f3f0c100210 2024-03-09T00:04:37.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.433+0000 7f3f027fc640 1 -- 172.21.15.73:0/2859320112 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7f3f0809b610 con 0x7f3f0c100210 2024-03-09T00:04:37.428 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:37.428 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 msgr2=0x7f3ef80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 0x7f3ef80a4cc0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f3efc006010 tx=0x7f3efc000bc0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 msgr2=0x7f3f0c1975e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f3f0800d950 tx=0x7f3f0800de10 comp rx=0 tx=0).stop 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 shutdown_connections 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3ef80a2800 0x7f3ef80a4cc0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3f0c101dc0 0x7f3f0c19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3f0c101400 0x7f3f0c197b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 --2- 172.21.15.73:0/2859320112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3f0c100210 0x7f3f0c1975e0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 >> 172.21.15.73:0/2859320112 conn(0x7f3f0c0fb780 msgr2=0x7f3f0c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:37.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 shutdown_connections 2024-03-09T00:04:37.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.435+0000 7f3f13390640 1 -- 172.21.15.73:0/2859320112 wait complete. 2024-03-09T00:04:37.441 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 -- 172.21.15.73:0/3551061248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28101400 msgr2=0x7fea28101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 --2- 172.21.15.73:0/3551061248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28101400 0x7fea28101880 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009900 tx=0x7fea1c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 -- 172.21.15.73:0/3551061248 shutdown_connections 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 --2- 172.21.15.73:0/3551061248 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101dc0 0x7fea28108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 --2- 172.21.15.73:0/3551061248 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28101400 0x7fea28101880 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 --2- 172.21.15.73:0/3551061248 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28100210 0x7fea28100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 -- 172.21.15.73:0/3551061248 >> 172.21.15.73:0/3551061248 conn(0x7fea280fb7a0 msgr2=0x7fea280fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 -- 172.21.15.73:0/3551061248 shutdown_connections 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.550+0000 7fea2d059640 1 -- 172.21.15.73:0/3551061248 wait complete. 2024-03-09T00:04:37.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 Processor -- start 2024-03-09T00:04:37.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 -- start start 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101400 0x7fea28197ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 0x7fea2819eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2810bea0 con 0x7fea28100210 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2810bd20 con 0x7fea28101dc0 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.551+0000 7fea2d059640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea2810c020 con 0x7fea28101400 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34760/0 (socket says 172.21.15.73:34760) 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 -- 172.21.15.73:0/2876879983 learned_addr learned my addr 172.21.15.73:0/2876879983 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea2759e640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 0x7fea2819eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea2659c640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101400 0x7fea28197ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 -- 172.21.15.73:0/2876879983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101400 msgr2=0x7fea28197ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101400 0x7fea28197ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 -- 172.21.15.73:0/2876879983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 msgr2=0x7fea2819eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 0x7fea2819eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 -- 172.21.15.73:0/2876879983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea2819f1b0 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea2759e640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 0x7fea2819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea26d9d640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c970 tx=0x7fea2000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea20007c10 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea20007db0 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea2819f4a0 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea2000f940 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.552+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea2819f9f0 con 0x7fea28100210 2024-03-09T00:04:37.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.553+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea2819fcb0 con 0x7fea28100210 2024-03-09T00:04:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.554+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fea2000fae0 con 0x7fea28100210 2024-03-09T00:04:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.555+0000 7fea0ffff640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 0x7fea100a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.555+0000 7fea2659c640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 0x7fea100a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.555+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fea200cf3f0 con 0x7fea28100210 2024-03-09T00:04:37.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.555+0000 7fea2659c640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 0x7fea100a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fea1c036040 tx=0x7fea1c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:37.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.558+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fea2009c370 con 0x7fea28100210 2024-03-09T00:04:37.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.770+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fea281a00e0 con 0x7fea28100210 2024-03-09T00:04:37.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.978+0000 7fea0ffff640 1 -- 172.21.15.73:0/2876879983 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7fea2009bc70 con 0x7fea28100210 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 msgr2=0x7fea100a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 0x7fea100a4cc0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fea1c036040 tx=0x7fea1c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 msgr2=0x7fea28197590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7fea2000c970 tx=0x7fea2000ce30 comp rx=0 tx=0).stop 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 shutdown_connections 2024-03-09T00:04:37.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fea100a2800 0x7fea100a4cc0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea28101dc0 0x7fea2819eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea28101400 0x7fea28197ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 --2- 172.21.15.73:0/2876879983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea28100210 0x7fea28197590 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.980+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 >> 172.21.15.73:0/2876879983 conn(0x7fea280fb7a0 msgr2=0x7fea280698c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.981+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 shutdown_connections 2024-03-09T00:04:37.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:37.981+0000 7fea2d059640 1 -- 172.21.15.73:0/2876879983 wait complete. 2024-03-09T00:04:37.987 DEBUG:tasks.cephfs.filesystem:fs_config: {'ec_profile': ['m=2', 'k=2', 'crush-failure-domain=osd'], 'max_mds': 1, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'}} 2024-03-09T00:04:37.987 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-09T00:04:38.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 -- 172.21.15.73:0/1543006624 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 msgr2=0x7f881c101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 --2- 172.21.15.73:0/1543006624 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c101730 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8818009900 tx=0x7f881802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 -- 172.21.15.73:0/1543006624 shutdown_connections 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 --2- 172.21.15.73:0/1543006624 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102ec0 0x7f881c109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 --2- 172.21.15.73:0/1543006624 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102500 0x7f881c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 --2- 172.21.15.73:0/1543006624 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c101730 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 -- 172.21.15.73:0/1543006624 >> 172.21.15.73:0/1543006624 conn(0x7f881c0fc880 msgr2=0x7f881c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 -- 172.21.15.73:0/1543006624 shutdown_connections 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 -- 172.21.15.73:0/1543006624 wait complete. 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.099+0000 7f8822c85640 1 Processor -- start 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 -- start start 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:38.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 0x7f881c0736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102ec0 0x7f881c073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c10cfe0 con 0x7f881c101310 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c10ce60 con 0x7f881c102500 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f881c10d160 con 0x7f881c102ec0 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34770/0 (socket says 172.21.15.73:34770) 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 -- 172.21.15.73:0/1595491353 learned_addr learned my addr 172.21.15.73:0/1595491353 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8813fff640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 0x7f881c0736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8821222640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102ec0 0x7f881c073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 -- 172.21.15.73:0/1595491353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102ec0 msgr2=0x7f881c073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102ec0 0x7f881c073c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 -- 172.21.15.73:0/1595491353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 msgr2=0x7f881c0736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 0x7f881c0736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 -- 172.21.15.73:0/1595491353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881c0744b0 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8813fff640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 0x7f881c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8820a21640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8818000c00 tx=0x7f8818004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8818041070 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f88180054e0 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881c1a1710 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.100+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f88180346a0 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.101+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f881c1a1c60 con 0x7f881c101310 2024-03-09T00:04:38.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.102+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f881c108630 con 0x7f881c101310 2024-03-09T00:04:38.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.102+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8818044070 con 0x7f881c101310 2024-03-09T00:04:38.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.103+0000 7f8811ffb640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:38.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.103+0000 7f8813fff640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:38.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.103+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f88180f0b40 con 0x7f881c101310 2024-03-09T00:04:38.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.103+0000 7f8813fff640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 0x7f88040a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f880c006fd0 tx=0x7f880c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:38.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.107+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f88180bd9a0 con 0x7f881c101310 2024-03-09T00:04:38.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:37 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2514099570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:37 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2859320112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:37 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-09T00:04:38.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:37 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2514099570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:37 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2859320112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:37 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-09T00:04:38.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.321+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f881c061ef0 con 0x7f881c101310 2024-03-09T00:04:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:37 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2514099570' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:37 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2859320112' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:37 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-09T00:04:38.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.980+0000 7f8811ffb640 1 -- 172.21.15.73:0/1595491353 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v18) v1 ==== 121+0+0 (secure 0 0 0) 0x7f8818003e20 con 0x7f881c101310 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 msgr2=0x7f88040a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 0x7f88040a4cc0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f880c006fd0 tx=0x7f880c008040 comp rx=0 tx=0).stop 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 msgr2=0x7f881c0750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f8818000c00 tx=0x7f8818004040 comp rx=0 tx=0).stop 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 shutdown_connections 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88040a2800 0x7f88040a4cc0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f881c102ec0 0x7f881c073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f881c102500 0x7f881c0736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 --2- 172.21.15.73:0/1595491353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f881c101310 0x7f881c0750a0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:38.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.982+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 >> 172.21.15.73:0/1595491353 conn(0x7f881c0fc880 msgr2=0x7f881c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:38.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.983+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 shutdown_connections 2024-03-09T00:04:38.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:38.983+0000 7f8822c85640 1 -- 172.21.15.73:0/1595491353 wait complete. 2024-03-09T00:04:38.991 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-09T00:04:39.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 -- 172.21.15.73:0/2141666032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac701024d0 msgr2=0x7fac70104940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:39.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 --2- 172.21.15.73:0/2141666032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac701024d0 0x7fac70104940 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fac60009900 tx=0x7fac6002bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:39.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 -- 172.21.15.73:0/2141666032 shutdown_connections 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 --2- 172.21.15.73:0/2141666032 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 0x7fac70105300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 --2- 172.21.15.73:0/2141666032 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac701024d0 0x7fac70104940 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 --2- 172.21.15.73:0/2141666032 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac700ffb60 0x7fac70101f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 -- 172.21.15.73:0/2141666032 >> 172.21.15.73:0/2141666032 conn(0x7fac700fb780 msgr2=0x7fac700fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 -- 172.21.15.73:0/2141666032 shutdown_connections 2024-03-09T00:04:39.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.101+0000 7fac773d4640 1 -- 172.21.15.73:0/2141666032 wait complete. 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 Processor -- start 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 -- start start 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac701024d0 0x7fac7019e120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 0x7fac701a5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac70113540 con 0x7fac700ffb60 2024-03-09T00:04:39.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac701133c0 con 0x7fac701024d0 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac773d4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac701136c0 con 0x7fac70104e80 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35458/0 (socket says 172.21.15.73:35458) 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 -- 172.21.15.73:0/1357870455 learned_addr learned my addr 172.21.15.73:0/1357870455 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75971640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 0x7fac701a5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:39.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 -- 172.21.15.73:0/1357870455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 msgr2=0x7fac701a5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 0x7fac701a5190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 -- 172.21.15.73:0/1357870455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac701024d0 msgr2=0x7fac7019e120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac701024d0 0x7fac7019e120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 -- 172.21.15.73:0/1357870455 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac701a5800 con 0x7fac700ffb60 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.102+0000 7fac75170640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00c890 tx=0x7fac6c00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.103+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c004220 con 0x7fac700ffb60 2024-03-09T00:04:39.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.103+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fac6c0043c0 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.103+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fac6c0049e0 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.103+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac701a5af0 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.103+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac701a5ff0 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.104+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac70068560 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.105+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fac6c0079b0 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.105+0000 7fac667fc640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 0x7fac5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.106+0000 7fac7496f640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 0x7fac5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.106+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fac6c0cee10 con 0x7fac700ffb60 2024-03-09T00:04:39.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.106+0000 7fac7496f640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 0x7fac5c0a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac60036040 tx=0x7fac60000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:39.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.110+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fac70068560 con 0x7fac700ffb60 2024-03-09T00:04:39.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:38 smithi102 ceph-mon[36106]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-09T00:04:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:38 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:39.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-09T00:04:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.319+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7fac70061fc0 con 0x7fac700ffb60 2024-03-09T00:04:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:38 smithi179 ceph-mon[36637]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-09T00:04:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:38 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:39.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-09T00:04:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:38 smithi073 ceph-mon[32309]: pgmap v306: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 426 B/s wr, 0 op/s 2024-03-09T00:04:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2876879983' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-09T00:04:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:38 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:39.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-09T00:04:39.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.990+0000 7fac667fc640 1 -- 172.21.15.73:0/1357870455 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v20) v1 ==== 115+0+0 (secure 0 0 0) 0x7fac6c014030 con 0x7fac700ffb60 2024-03-09T00:04:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 msgr2=0x7fac5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 0x7fac5c0a4cc0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7fac60036040 tx=0x7fac60000bc0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 msgr2=0x7fac7019dbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fac6c00c890 tx=0x7fac6c00cd50 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 shutdown_connections 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fac5c0a2800 0x7fac5c0a4cc0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fac70104e80 0x7fac701a5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fac701024d0 0x7fac7019e120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 --2- 172.21.15.73:0/1357870455 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fac700ffb60 0x7fac7019dbe0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.993+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 >> 172.21.15.73:0/1357870455 conn(0x7fac700fb780 msgr2=0x7fac701004e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.994+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 shutdown_connections 2024-03-09T00:04:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:39.994+0000 7fac773d4640 1 -- 172.21.15.73:0/1357870455 wait complete. 2024-03-09T00:04:40.001 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2, 'subvol_options': '--namespace-isolated --size 25000000000'} subvols for filesystem 'cephfs' 2024-03-09T00:04:40.002 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 -- 172.21.15.73:0/889840719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 msgr2=0x7fb78c108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 --2- 172.21.15.73:0/889840719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c108630 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fb784009900 tx=0x7fb78402bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 -- 172.21.15.73:0/889840719 shutdown_connections 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 --2- 172.21.15.73:0/889840719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c108630 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 --2- 172.21.15.73:0/889840719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1013e0 0x7fb78c101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 --2- 172.21.15.73:0/889840719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1001f0 0x7fb78c100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 -- 172.21.15.73:0/889840719 >> 172.21.15.73:0/889840719 conn(0x7fb78c0fb780 msgr2=0x7fb78c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 -- 172.21.15.73:0/889840719 shutdown_connections 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.113+0000 7fb7914d1640 1 -- 172.21.15.73:0/889840719 wait complete. 2024-03-09T00:04:40.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 Processor -- start 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 -- start start 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1001f0 0x7fb78c1975d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1013e0 0x7fb78c197b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c10be80 con 0x7fb78c101da0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c10bd00 con 0x7fb78c1001f0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb7914d1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb78c10c000 con 0x7fb78c1013e0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb78b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb78b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35484/0 (socket says 172.21.15.73:35484) 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.114+0000 7fb78b7fe640 1 -- 172.21.15.73:0/1344758145 learned_addr learned my addr 172.21.15.73:0/1344758145 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78affd640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1001f0 0x7fb78c1975d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78a7fc640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1013e0 0x7fb78c197b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 -- 172.21.15.73:0/1344758145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1013e0 msgr2=0x7fb78c197b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1013e0 0x7fb78c197b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 -- 172.21.15.73:0/1344758145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1001f0 msgr2=0x7fb78c1975d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1001f0 0x7fb78c1975d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 -- 172.21.15.73:0/1344758145 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb78c19f1f0 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb78b7fe640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb78402ce70 tx=0x7fb784005e40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb784041070 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb78c19f480 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7840054e0 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7840344a0 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.115+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb78c19f970 con 0x7fb78c101da0 2024-03-09T00:04:40.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.116+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb78c19fbd0 con 0x7fb78c101da0 2024-03-09T00:04:40.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.117+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb784044070 con 0x7fb78c101da0 2024-03-09T00:04:40.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.118+0000 7fb773fff640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 0x7fb7740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.118+0000 7fb78affd640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 0x7fb7740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.118+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb7840f0db0 con 0x7fb78c101da0 2024-03-09T00:04:40.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.118+0000 7fb78affd640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 0x7fb7740a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb78c101240 tx=0x7fb77c00a590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.121+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb7840bdc10 con 0x7fb78c101da0 2024-03-09T00:04:40.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-09T00:04:40.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:39 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:40.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:39 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:40.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-09T00:04:40.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.325+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb78c0fd8f0 con 0x7fb7740a2800 2024-03-09T00:04:40.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-09T00:04:40.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:39 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:40.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:39 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:40.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-09T00:04:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1595491353' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-09T00:04:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:39 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-09T00:04:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:39 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-09T00:04:40.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.343+0000 7fb773fff640 1 -- 172.21.15.73:0/1344758145 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb78c0fd8f0 con 0x7fb7740a2800 2024-03-09T00:04:40.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 msgr2=0x7fb7740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 0x7fb7740a4cc0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7fb78c101240 tx=0x7fb77c00a590 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 msgr2=0x7fb78c19eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fb78402ce70 tx=0x7fb784005e40 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 shutdown_connections 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb7740a2800 0x7fb7740a4cc0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb78c101da0 0x7fb78c19eb80 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb78c1013e0 0x7fb78c197b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 --2- 172.21.15.73:0/1344758145 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb78c1001f0 0x7fb78c1975d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 >> 172.21.15.73:0/1344758145 conn(0x7fb78c0fb780 msgr2=0x7fb78c0fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.345+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 shutdown_connections 2024-03-09T00:04:40.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.346+0000 7fb7914d1640 1 -- 172.21.15.73:0/1344758145 wait complete. 2024-03-09T00:04:40.352 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- 172.21.15.73:0/3870811635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c100210 msgr2=0x7fdb5c100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 --2- 172.21.15.73:0/3870811635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c100210 0x7fdb5c100630 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c009900 tx=0x7fdb4c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- 172.21.15.73:0/3870811635 shutdown_connections 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 --2- 172.21.15.73:0/3870811635 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 --2- 172.21.15.73:0/3870811635 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 0x7fdb5c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 --2- 172.21.15.73:0/3870811635 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c100210 0x7fdb5c100630 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- 172.21.15.73:0/3870811635 >> 172.21.15.73:0/3870811635 conn(0x7fdb5c0fb7a0 msgr2=0x7fdb5c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:40.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- 172.21.15.73:0/3870811635 shutdown_connections 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- 172.21.15.73:0/3870811635 wait complete. 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 Processor -- start 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.464+0000 7fdb62545640 1 -- start start 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c100210 0x7fdb5c06a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 0x7fdb5c19c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c10bea0 con 0x7fdb5c101dc0 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c10bd20 con 0x7fdb5c100210 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb5c10c020 con 0x7fdb5c101400 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35536/0 (socket says 172.21.15.73:35536) 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 -- 172.21.15.73:0/186458322 learned_addr learned my addr 172.21.15.73:0/186458322 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:40.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb5bfff640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c100210 0x7fdb5c06a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb5b7fe640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 0x7fdb5c19c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 -- 172.21.15.73:0/186458322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 msgr2=0x7fdb5c19c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 0x7fdb5c19c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 -- 172.21.15.73:0/186458322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c100210 msgr2=0x7fdb5c06a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c100210 0x7fdb5c06a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 -- 172.21.15.73:0/186458322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5c19f2b0 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb60ae2640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400eaa0 tx=0x7fdb5400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb5400cdb0 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb54005020 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb5c19f540 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb540116b0 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.465+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb5c19fa40 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.466+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb5c1075b0 con 0x7fdb5c101dc0 2024-03-09T00:04:40.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.467+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fdb54005560 con 0x7fdb5c101dc0 2024-03-09T00:04:40.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.468+0000 7fdb597fa640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 0x7fdb440a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.468+0000 7fdb5bfff640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 0x7fdb440a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.468+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdb540cee50 con 0x7fdb5c101dc0 2024-03-09T00:04:40.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.468+0000 7fdb5bfff640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 0x7fdb440a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c000c00 tx=0x7fdb4c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.472+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdb54016030 con 0x7fdb5c101dc0 2024-03-09T00:04:40.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:04:40.330+0000 7fb397fea700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:04:40.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:04:40.330+0000 7fb397fea700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:04:40.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:04:40.330+0000 7fb397fea700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:04:40.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:04:40.330+0000 7fb397fea700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:04:40.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:04:40.330+0000 7fb397fea700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:04:40.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.674+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb5c0617c0 con 0x7fdb440a2800 2024-03-09T00:04:40.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.695+0000 7fdb597fa640 1 -- 172.21.15.73:0/186458322 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fdb5c0617c0 con 0x7fdb440a2800 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 msgr2=0x7fdb440a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 0x7fdb440a4cc0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fdb4c000c00 tx=0x7fdb4c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 msgr2=0x7fdb5c19e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fdb5400eaa0 tx=0x7fdb5400ef60 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 shutdown_connections 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb440a2800 0x7fdb440a4cc0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb5c101dc0 0x7fdb5c19e9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb5c101400 0x7fdb5c19c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 --2- 172.21.15.73:0/186458322 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb5c100210 0x7fdb5c06a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 >> 172.21.15.73:0/186458322 conn(0x7fdb5c0fb7a0 msgr2=0x7fdb5c104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 shutdown_connections 2024-03-09T00:04:40.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.697+0000 7fdb62545640 1 -- 172.21.15.73:0/186458322 wait complete. 2024-03-09T00:04:40.703 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 -- 172.21.15.73:0/3405032076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c100ea0 msgr2=0x7f723c108740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 --2- 172.21.15.73:0/3405032076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c100ea0 0x7f723c108740 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f722c009900 tx=0x7f722c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 -- 172.21.15.73:0/3405032076 shutdown_connections 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 --2- 172.21.15.73:0/3405032076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c100ea0 0x7f723c108740 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 --2- 172.21.15.73:0/3405032076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 0x7f723c100960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 --2- 172.21.15.73:0/3405032076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c1017b0 0x7f723c101bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 -- 172.21.15.73:0/3405032076 >> 172.21.15.73:0/3405032076 conn(0x7f723c0faf10 msgr2=0x7f723c0fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 -- 172.21.15.73:0/3405032076 shutdown_connections 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.815+0000 7f7240e97640 1 -- 172.21.15.73:0/3405032076 wait complete. 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 Processor -- start 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 -- start start 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 0x7f723c197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c100ea0 0x7f723c197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c10d020 con 0x7f723c1017b0 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c10cea0 con 0x7f723c1004e0 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7240e97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f723c10d1a0 con 0x7f723c100ea0 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35564/0 (socket says 172.21.15.73:35564) 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723ad9d640 1 -- 172.21.15.73:0/1248043652 learned_addr learned my addr 172.21.15.73:0/1248043652 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f7239d9b640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c100ea0 0x7f723c197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723a59c640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 0x7f723c197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723ad9d640 1 -- 172.21.15.73:0/1248043652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c100ea0 msgr2=0x7f723c197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.816+0000 7f723ad9d640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c100ea0 0x7f723c197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f723ad9d640 1 -- 172.21.15.73:0/1248043652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 msgr2=0x7f723c197650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f723ad9d640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 0x7f723c197650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f723ad9d640 1 -- 172.21.15.73:0/1248043652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c19f270 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f723ad9d640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f722c005ab0 tx=0x7f722c004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c041070 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f723c19f500 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f722c0054e0 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f722c0346a0 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.817+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f723c19f9f0 con 0x7f723c1017b0 2024-03-09T00:04:40.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.818+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f723c107550 con 0x7f723c1017b0 2024-03-09T00:04:40.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.819+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 16) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f722c044070 con 0x7f723c1017b0 2024-03-09T00:04:40.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.819+0000 7f722b7fe640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 0x7f72240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:40.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.820+0000 7f723a59c640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 0x7f72240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:40.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.820+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f722c0f0bd0 con 0x7f723c1017b0 2024-03-09T00:04:40.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.820+0000 7f723a59c640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 0x7f72240a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f723c069700 tx=0x7f723400d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:40.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:40.824+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f722c0bda30 con 0x7f723c1017b0 2024-03-09T00:04:41.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.006+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f722c0b99a0 con 0x7f723c1017b0 2024-03-09T00:04:41.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.032+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f723c0617c0 con 0x7f72240a2800 2024-03-09T00:04:41.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.036+0000 7f722b7fe640 1 -- 172.21.15.73:0/1248043652 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f723c0617c0 con 0x7f72240a2800 2024-03-09T00:04:41.033 INFO:teuthology.orchestra.run.smithi073.stdout:/volumes/qa/sv_0/5cd7948f-5f1a-4b63-9f76-7e67bd8833df 2024-03-09T00:04:41.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.037+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 msgr2=0x7f72240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.037+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 0x7f72240a4cc0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f723c069700 tx=0x7f723400d040 comp rx=0 tx=0).stop 2024-03-09T00:04:41.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.037+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 msgr2=0x7f723c19ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.037+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f722c005ab0 tx=0x7f722c004270 comp rx=0 tx=0).stop 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 shutdown_connections 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f72240a2800 0x7f72240a4cc0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f723c1017b0 0x7f723c19ec00 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f723c100ea0 0x7f723c197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 --2- 172.21.15.73:0/1248043652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f723c1004e0 0x7f723c197650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 >> 172.21.15.73:0/1248043652 conn(0x7f723c0faf10 msgr2=0x7f723c104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 shutdown_connections 2024-03-09T00:04:41.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.038+0000 7f7240e97640 1 -- 172.21.15.73:0/1248043652 wait complete. 2024-03-09T00:04:41.044 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-09T00:04:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- 172.21.15.73:0/1227126182 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 msgr2=0x7fb2a0102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- 172.21.15.73:0/1227126182 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0102970 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009900 tx=0x7fb29c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- 172.21.15.73:0/1227126182 shutdown_connections 2024-03-09T00:04:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- 172.21.15.73:0/1227126182 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0102eb0 0x7fb2a0109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- 172.21.15.73:0/1227126182 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0102970 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- 172.21.15.73:0/1227126182 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0101300 0x7fb2a0101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- 172.21.15.73:0/1227126182 >> 172.21.15.73:0/1227126182 conn(0x7fb2a00fc890 msgr2=0x7fb2a00fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- 172.21.15.73:0/1227126182 shutdown_connections 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- 172.21.15.73:0/1227126182 wait complete. 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 Processor -- start 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- start start 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0101300 0x7fb2a0197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 0x7fb2a019ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a010cff0 con 0x7fb2a01024f0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a010ce70 con 0x7fb2a0101300 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.156+0000 7fb2a7909640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2a010d170 con 0x7fb2a0102eb0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35578/0 (socket says 172.21.15.73:35578) 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 -- 172.21.15.73:0/3319143152 learned_addr learned my addr 172.21.15.73:0/3319143152 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a56a5640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0101300 0x7fb2a0197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a5ea6640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 0x7fb2a019ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 -- 172.21.15.73:0/3319143152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 msgr2=0x7fb2a019ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 0x7fb2a019ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 -- 172.21.15.73:0/3319143152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0101300 msgr2=0x7fb2a0197690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0101300 0x7fb2a0197690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 -- 172.21.15.73:0/3319143152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2a019f2b0 con 0x7fb2a01024f0 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a5ea6640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 0x7fb2a019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a4ea4640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009230 tx=0x7fb29c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c041070 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2a019f540 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb29c0054e0 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb29c034440 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.157+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2a019fa80 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.158+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2a01086a0 con 0x7fb2a01024f0 2024-03-09T00:04:41.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.159+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fb29c044070 con 0x7fb2a01024f0 2024-03-09T00:04:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.160+0000 7fb2927fc640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 0x7fb28c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.160+0000 7fb2a56a5640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 0x7fb28c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.160+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fb29c0f0ea0 con 0x7fb2a01024f0 2024-03-09T00:04:41.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.160+0000 7fb2a56a5640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 0x7fb28c0a4d50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0102350 tx=0x7fb294009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.164+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb29c0bdc50 con 0x7fb2a01024f0 2024-03-09T00:04:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:40 smithi102 ceph-mon[36106]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-09T00:04:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:40 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-09T00:04:41.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:40 smithi102 ceph-mon[36106]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:41.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:40 smithi179 ceph-mon[36637]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-09T00:04:41.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:40 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-09T00:04:41.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:40 smithi179 ceph-mon[36637]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:41.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.368+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2a00617c0 con 0x7fb28c0a2890 2024-03-09T00:04:41.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.389+0000 7fb2927fc640 1 -- 172.21.15.73:0/3319143152 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fb2a00617c0 con 0x7fb28c0a2890 2024-03-09T00:04:41.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 msgr2=0x7fb28c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 0x7fb28c0a4d50 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fb2a0102350 tx=0x7fb294009110 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 msgr2=0x7fb2a0197bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fb29c009230 tx=0x7fb29c005dd0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 shutdown_connections 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb28c0a2890 0x7fb28c0a4d50 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb2a0102eb0 0x7fb2a019ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb2a01024f0 0x7fb2a0197bd0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 --2- 172.21.15.73:0/3319143152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb2a0101300 0x7fb2a0197690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 >> 172.21.15.73:0/3319143152 conn(0x7fb2a00fc890 msgr2=0x7fb2a0105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 shutdown_connections 2024-03-09T00:04:41.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.391+0000 7fb2a7909640 1 -- 172.21.15.73:0/3319143152 wait complete. 2024-03-09T00:04:41.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-mon[32309]: pgmap v307: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 597 B/s rd, 426 B/s wr, 0 op/s 2024-03-09T00:04:41.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1357870455' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-09T00:04:41.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:40 smithi073 ceph-mon[32309]: fsmap cephfs:1 {0=b=up:active} 1 up:standby-replay 10 up:standby 2024-03-09T00:04:41.397 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-09T00:04:41.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 -- 172.21.15.73:0/46971689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101400 msgr2=0x7fc0d4101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/46971689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101400 0x7fc0d4101880 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc0c8009900 tx=0x7fc0c802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:41.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 -- 172.21.15.73:0/46971689 shutdown_connections 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/46971689 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/46971689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101400 0x7fc0d4101880 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/46971689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 0x7fc0d4100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 -- 172.21.15.73:0/46971689 >> 172.21.15.73:0/46971689 conn(0x7fc0d40fb7c0 msgr2=0x7fc0d40fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 -- 172.21.15.73:0/46971689 shutdown_connections 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.508+0000 7fc0d97d4640 1 -- 172.21.15.73:0/46971689 wait complete. 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 Processor -- start 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 -- start start 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 0x7fc0d41041a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101400 0x7fc0d41027e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d410bea0 con 0x7fc0d4101dc0 2024-03-09T00:04:41.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d410bd20 con 0x7fc0d4100210 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d97d4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d410c020 con 0x7fc0d4101400 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35590/0 (socket says 172.21.15.73:35590) 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d37fe640 1 -- 172.21.15.73:0/3532259866 learned_addr learned my addr 172.21.15.73:0/3532259866 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d2ffd640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 0x7fc0d41041a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.509+0000 7fc0d27fc640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101400 0x7fc0d41027e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 -- 172.21.15.73:0/3532259866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101400 msgr2=0x7fc0d41027e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101400 0x7fc0d41027e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 -- 172.21.15.73:0/3532259866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 msgr2=0x7fc0d41041a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 0x7fc0d41041a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 -- 172.21.15.73:0/3532259866 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d41035b0 con 0x7fc0d4101dc0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d37fe640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e970 tx=0x7fc0cc00ed70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc00ccc0 con 0x7fc0d4101dc0 2024-03-09T00:04:41.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0d41a16b0 con 0x7fc0d4101dc0 2024-03-09T00:04:41.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0cc00ce60 con 0x7fc0d4101dc0 2024-03-09T00:04:41.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0cc010490 con 0x7fc0d4101dc0 2024-03-09T00:04:41.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.510+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d41a1b80 con 0x7fc0d4101dc0 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.511+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d4068530 con 0x7fc0d4101dc0 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.512+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc0cc005020 con 0x7fc0d4101dc0 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.512+0000 7fc0bbfff640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 0x7fc0bc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.513+0000 7fc0d2ffd640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 0x7fc0bc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.513+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fc0cc0cf230 con 0x7fc0d4101dc0 2024-03-09T00:04:41.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.513+0000 7fc0d2ffd640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 0x7fc0bc0a4d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4101260 tx=0x7fc0c4009670 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.517+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc0cc09bfd0 con 0x7fc0d4101dc0 2024-03-09T00:04:41.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.719+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fc0d40fd910 con 0x7fc0bc0a2890 2024-03-09T00:04:41.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.722+0000 7fc0bbfff640 1 -- 172.21.15.73:0/3532259866 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7fc0d40fd910 con 0x7fc0bc0a2890 2024-03-09T00:04:41.717 INFO:teuthology.orchestra.run.smithi073.stdout:/volumes/qa/sv_1/e1c20de2-a1eb-4247-9eda-1427c89645a8 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 msgr2=0x7fc0bc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 0x7fc0bc0a4d50 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fc0d4101260 tx=0x7fc0c4009670 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 msgr2=0x7fc0d4102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fc0cc00e970 tx=0x7fc0cc00ed70 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 shutdown_connections 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc0bc0a2890 0x7fc0bc0a4d50 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0d4101dc0 0x7fc0d4102d20 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0d4101400 0x7fc0d41027e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 --2- 172.21.15.73:0/3532259866 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0d4100210 0x7fc0d41041a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.724+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 >> 172.21.15.73:0/3532259866 conn(0x7fc0d40fb7c0 msgr2=0x7fc0d40fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.725+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 shutdown_connections 2024-03-09T00:04:41.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.725+0000 7fc0d97d4640 1 -- 172.21.15.73:0/3532259866 wait complete. 2024-03-09T00:04:41.731 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 -- 172.21.15.73:0/3021541200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394101dc0 msgr2=0x7f2394108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 --2- 172.21.15.73:0/3021541200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394101dc0 0x7f2394108650 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f238c009900 tx=0x7f238c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 -- 172.21.15.73:0/3021541200 shutdown_connections 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 --2- 172.21.15.73:0/3021541200 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394101dc0 0x7f2394108650 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 --2- 172.21.15.73:0/3021541200 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f2394101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 --2- 172.21.15.73:0/3021541200 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394100210 0x7f2394100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.837+0000 7f239c68c640 1 -- 172.21.15.73:0/3021541200 >> 172.21.15.73:0/3021541200 conn(0x7f23940fb780 msgr2=0x7f23940fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- 172.21.15.73:0/3021541200 shutdown_connections 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- 172.21.15.73:0/3021541200 wait complete. 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 Processor -- start 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- start start 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f239419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394101dc0 0x7f239419e940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239410bea0 con 0x7f2394100210 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239410bd20 con 0x7f2394101dc0 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239c68c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f239410c020 con 0x7f2394101400 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239a428640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239a428640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35600/0 (socket says 172.21.15.73:35600) 2024-03-09T00:04:41.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239a428640 1 -- 172.21.15.73:0/1326044312 learned_addr learned my addr 172.21.15.73:0/1326044312 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f239ac29640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394101dc0 0x7f239419e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.838+0000 7f2399c27640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f239419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 -- 172.21.15.73:0/1326044312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 msgr2=0x7f239419c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f239419c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 -- 172.21.15.73:0/1326044312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394101dc0 msgr2=0x7f239419e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394101dc0 0x7f239419e940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 -- 172.21.15.73:0/1326044312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239419f200 con 0x7f2394100210 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f2399c27640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f239419c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239a428640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f239000ea80 tx=0x7f239000ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239000ce40 con 0x7f2394100210 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2390011950 con 0x7f2394100210 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f239419f4f0 con 0x7f2394100210 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f239001ab80 con 0x7f2394100210 2024-03-09T00:04:41.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.839+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f239419fa40 con 0x7f2394100210 2024-03-09T00:04:41.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.840+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2394107530 con 0x7f2394100210 2024-03-09T00:04:41.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.841+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2390011b00 con 0x7f2394100210 2024-03-09T00:04:41.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.842+0000 7f238b7fe640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 0x7f23800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:41.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.842+0000 7f2399c27640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 0x7f23800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:41.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.842+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f23900cf4b0 con 0x7f2394100210 2024-03-09T00:04:41.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.842+0000 7f2399c27640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 0x7f23800a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f23840058c0 tx=0x7f2384005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:41.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:41.846+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2390016030 con 0x7f2394100210 2024-03-09T00:04:42.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.061+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f239402d040 con 0x7f2394100210 2024-03-09T00:04:42.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.062+0000 7f238b7fe640 1 -- 172.21.15.73:0/1326044312 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f239009c1b0 con 0x7f2394100210 2024-03-09T00:04:42.057 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:42.057 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:04:42.057 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 msgr2=0x7f23800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 0x7f23800a4d50 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f23840058c0 tx=0x7f2384005850 comp rx=0 tx=0).stop 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 msgr2=0x7f2394068fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f239000ea80 tx=0x7f239000ef40 comp rx=0 tx=0).stop 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 shutdown_connections 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23800a2890 0x7f23800a4d50 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2394101dc0 0x7f239419e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2394101400 0x7f239419c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 --2- 172.21.15.73:0/1326044312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2394100210 0x7f2394068fe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 >> 172.21.15.73:0/1326044312 conn(0x7f23940fb780 msgr2=0x7f2394104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 shutdown_connections 2024-03-09T00:04:42.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.064+0000 7f239c68c640 1 -- 172.21.15.73:0/1326044312 wait complete. 2024-03-09T00:04:42.070 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- 172.21.15.73:0/1620587049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178101dc0 msgr2=0x7fa178108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/1620587049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178101dc0 0x7fa178108650 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fa170009ca0 tx=0x7fa17002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- 172.21.15.73:0/1620587049 shutdown_connections 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/1620587049 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178101dc0 0x7fa178108650 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/1620587049 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101400 0x7fa178101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/1620587049 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178100210 0x7fa178100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- 172.21.15.73:0/1620587049 >> 172.21.15.73:0/1620587049 conn(0x7fa1780fb7a0 msgr2=0x7fa1780fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- 172.21.15.73:0/1620587049 shutdown_connections 2024-03-09T00:04:42.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- 172.21.15.73:0/1620587049 wait complete. 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 Processor -- start 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 -- start start 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.177+0000 7fa17e2c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178101400 0x7fa17819c370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 0x7fa17819e990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17810bf00 con 0x7fa178100210 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17810bd80 con 0x7fa178101dc0 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17810c080 con 0x7fa178101400 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35608/0 (socket says 172.21.15.73:35608) 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 -- 172.21.15.73:0/3851581623 learned_addr learned my addr 172.21.15.73:0/3851581623 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa1777fe640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178101400 0x7fa17819c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17c85d640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 0x7fa17819e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 -- 172.21.15.73:0/3851581623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178101400 msgr2=0x7fa17819c370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178101400 0x7fa17819c370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 -- 172.21.15.73:0/3851581623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 msgr2=0x7fa17819e990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 0x7fa17819e990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 -- 172.21.15.73:0/3851581623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17819f2a0 con 0x7fa178100210 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17c85d640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 0x7fa17819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa177fff640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c970 tx=0x7fa16800cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168004220 con 0x7fa178100210 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1680043c0 con 0x7fa178100210 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa17819f530 con 0x7fa178100210 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa168004a00 con 0x7fa178100210 2024-03-09T00:04:42.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.178+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa17819fa30 con 0x7fa178100210 2024-03-09T00:04:42.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.179+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa178107530 con 0x7fa178100210 2024-03-09T00:04:42.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.180+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fa1680079b0 con 0x7fa178100210 2024-03-09T00:04:42.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.181+0000 7fa1757fa640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 0x7fa1600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.181+0000 7fa1777fe640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 0x7fa1600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.181+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fa1680cebf0 con 0x7fa178100210 2024-03-09T00:04:42.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.181+0000 7fa1777fe640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 0x7fa1600a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa17806a220 tx=0x7fa16c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.185+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa168014030 con 0x7fa178100210 2024-03-09T00:04:42.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:41 smithi102 ceph-mon[36106]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:41 smithi102 ceph-mon[36106]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:41 smithi102 ceph-mon[36106]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-09T00:04:42.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:41 smithi102 ceph-mon[36106]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:41 smithi179 ceph-mon[36637]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:41 smithi179 ceph-mon[36637]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:41 smithi179 ceph-mon[36637]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-09T00:04:42.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:41 smithi179 ceph-mon[36637]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.362+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa17802d0c0 con 0x7fa178100210 2024-03-09T00:04:42.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.363+0000 7fa1757fa640 1 -- 172.21.15.73:0/3851581623 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7fa16809b8f0 con 0x7fa178100210 2024-03-09T00:04:42.359 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:42.359 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.365+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 msgr2=0x7fa1600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.365+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 0x7fa1600a4d50 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa17806a220 tx=0x7fa16c009110 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.365+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 msgr2=0x7fa17806a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.365+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fa16800c970 tx=0x7fa16800cd70 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 shutdown_connections 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa1600a2890 0x7fa1600a4d50 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa178101dc0 0x7fa17819e990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa178101400 0x7fa17819c370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 --2- 172.21.15.73:0/3851581623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa178100210 0x7fa17806a4b0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 >> 172.21.15.73:0/3851581623 conn(0x7fa1780fb7a0 msgr2=0x7fa178104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 shutdown_connections 2024-03-09T00:04:42.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.366+0000 7fa17e2c0640 1 -- 172.21.15.73:0/3851581623 wait complete. 2024-03-09T00:04:42.370 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:41 smithi073 ceph-mon[32309]: from='client.14892 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.370 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:41 smithi073 ceph-mon[32309]: from='client.14904 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.370 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:41 smithi073 ceph-mon[32309]: mgrmap e17: x(active, since 7m), standbys: z, y 2024-03-09T00:04:42.371 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:41 smithi073 ceph-mon[32309]: from='client.14910 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:42.372 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-09T00:04:42.372 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 -- 172.21.15.73:0/3601251520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4101480 msgr2=0x7f8ad4101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/3601251520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4101480 0x7f8ad4101900 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f8ac8009900 tx=0x7f8ac802bc30 comp rx=0 tx=0).stop 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 -- 172.21.15.73:0/3601251520 shutdown_connections 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/3601251520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101e40 0x7f8ad41086d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/3601251520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4101480 0x7f8ad4101900 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/3601251520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4100290 0x7f8ad41006b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 -- 172.21.15.73:0/3601251520 >> 172.21.15.73:0/3601251520 conn(0x7f8ad40fb840 msgr2=0x7f8ad40fdc60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 -- 172.21.15.73:0/3601251520 shutdown_connections 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.478+0000 7f8adb1f1640 1 -- 172.21.15.73:0/3601251520 wait complete. 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 Processor -- start 2024-03-09T00:04:42.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 -- start start 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101480 0x7f8ad419c300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 0x7f8ad419e920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad410beb0 con 0x7f8ad4100290 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad410bd30 con 0x7f8ad4101e40 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8adb1f1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad410c030 con 0x7f8ad4101480 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35630/0 (socket says 172.21.15.73:35630) 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 -- 172.21.15.73:0/2212145615 learned_addr learned my addr 172.21.15.73:0/2212145615 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad3fff640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101480 0x7f8ad419c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad978e640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 0x7f8ad419e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 -- 172.21.15.73:0/2212145615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101480 msgr2=0x7f8ad419c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101480 0x7f8ad419c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 -- 172.21.15.73:0/2212145615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 msgr2=0x7f8ad419e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 0x7f8ad419e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad8f8d640 1 -- 172.21.15.73:0/2212145615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad419f230 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.479+0000 7f8ad978e640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 0x7f8ad419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8ad8f8d640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c970 tx=0x7f8ac400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4004220 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ac40043c0 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac400f450 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad419f4c0 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.480+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad419f9e0 con 0x7f8ad4100290 2024-03-09T00:04:42.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.481+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ad4068530 con 0x7f8ad4100290 2024-03-09T00:04:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.482+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8ac40079b0 con 0x7f8ad4100290 2024-03-09T00:04:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.482+0000 7f8ad1ffb640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 0x7f8abc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.482+0000 7f8ad3fff640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 0x7f8abc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.482+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8ac40cebb0 con 0x7f8ad4100290 2024-03-09T00:04:42.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.483+0000 7f8ad3fff640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 0x7f8abc0a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ac802c7b0 tx=0x7f8ac8005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.487+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8ac4014030 con 0x7f8ad4100290 2024-03-09T00:04:42.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.657+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8ad40617f0 con 0x7f8abc0a2890 2024-03-09T00:04:42.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.667+0000 7f8ad1ffb640 1 -- 172.21.15.73:0/2212145615 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340206 (secure 0 0 0) 0x7f8ad40617f0 con 0x7f8abc0a2890 2024-03-09T00:04:42.667 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:42.669 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 msgr2=0x7f8abc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 0x7f8abc0a4d50 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8ac802c7b0 tx=0x7f8ac8005fd0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 msgr2=0x7f8ad406a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f8ac400c970 tx=0x7f8ac400ce30 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 shutdown_connections 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8abc0a2890 0x7f8abc0a4d50 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad4101e40 0x7f8ad419e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad4101480 0x7f8ad419c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 --2- 172.21.15.73:0/2212145615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad4100290 0x7f8ad406a4b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 >> 172.21.15.73:0/2212145615 conn(0x7f8ad40fb840 msgr2=0x7f8ad4104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 shutdown_connections 2024-03-09T00:04:42.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.671+0000 7f8adb1f1640 1 -- 172.21.15.73:0/2212145615 wait complete. 2024-03-09T00:04:42.677 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-03-09T00:04:41.919495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595125,"num_objects":25,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":231,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387544,"kb_used_data":13360,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554280,"statfs":{"total":1151940427776,"available":1151543582720,"internally_reserved":0,"allocated":13680640,"data_stored":7515700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285599+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:04:19.285599+0000","last_peered":"2024-03-09T00:04:19.285599+0000","last_clean":"2024-03-09T00:04:19.285599+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:04:19.285599+0000","last_undegraded":"2024-03-09T00:04:19.285599+0000","last_fullsized":"2024-03-09T00:04:19.285599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940819+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:04:18.940819+0000","last_peered":"2024-03-09T00:04:18.940819+0000","last_clean":"2024-03-09T00:04:18.940819+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:04:18.940819+0000","last_undegraded":"2024-03-09T00:04:18.940819+0000","last_fullsized":"2024-03-09T00:04:18.940819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940687+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:04:18.940687+0000","last_peered":"2024-03-09T00:04:18.940687+0000","last_clean":"2024-03-09T00:04:18.940687+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:04:18.940687+0000","last_undegraded":"2024-03-09T00:04:18.940687+0000","last_fullsized":"2024-03-09T00:04:18.940687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284869+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:04:19.284869+0000","last_peered":"2024-03-09T00:04:19.284869+0000","last_clean":"2024-03-09T00:04:19.284869+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:04:19.284869+0000","last_undegraded":"2024-03-09T00:04:19.284869+0000","last_fullsized":"2024-03-09T00:04:19.284869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284695+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:04:19.284695+0000","last_peered":"2024-03-09T00:04:19.284695+0000","last_clean":"2024-03-09T00:04:19.284695+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:04:19.284695+0000","last_undegraded":"2024-03-09T00:04:19.284695+0000","last_fullsized":"2024-03-09T00:04:19.284695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938353+0000","last_change":"2024-03-09T00:04:13.929878+0000","last_active":"2024-03-09T00:04:18.938353+0000","last_peered":"2024-03-09T00:04:18.938353+0000","last_clean":"2024-03-09T00:04:18.938353+0000","last_became_active":"2024-03-09T00:04:13.929508+0000","last_became_peered":"2024-03-09T00:04:13.929508+0000","last_unstale":"2024-03-09T00:04:18.938353+0000","last_undegraded":"2024-03-09T00:04:18.938353+0000","last_fullsized":"2024-03-09T00:04:18.938353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:51:07.490539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116022+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:04:19.116022+0000","last_peered":"2024-03-09T00:04:19.116022+0000","last_clean":"2024-03-09T00:04:19.116022+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:04:19.116022+0000","last_undegraded":"2024-03-09T00:04:19.116022+0000","last_fullsized":"2024-03-09T00:04:19.116022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.283878+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:04:19.283878+0000","last_peered":"2024-03-09T00:04:19.283878+0000","last_clean":"2024-03-09T00:04:19.283878+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:04:19.283878+0000","last_undegraded":"2024-03-09T00:04:19.283878+0000","last_fullsized":"2024-03-09T00:04:19.283878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285014+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:04:19.285014+0000","last_peered":"2024-03-09T00:04:19.285014+0000","last_clean":"2024-03-09T00:04:19.285014+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:04:19.285014+0000","last_undegraded":"2024-03-09T00:04:19.285014+0000","last_fullsized":"2024-03-09T00:04:19.285014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938817+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:04:18.938817+0000","last_peered":"2024-03-09T00:04:18.938817+0000","last_clean":"2024-03-09T00:04:18.938817+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:04:18.938817+0000","last_undegraded":"2024-03-09T00:04:18.938817+0000","last_fullsized":"2024-03-09T00:04:18.938817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116608+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:04:19.116608+0000","last_peered":"2024-03-09T00:04:19.116608+0000","last_clean":"2024-03-09T00:04:19.116608+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:04:19.116608+0000","last_undegraded":"2024-03-09T00:04:19.116608+0000","last_fullsized":"2024-03-09T00:04:19.116608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284763+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:04:19.284763+0000","last_peered":"2024-03-09T00:04:19.284763+0000","last_clean":"2024-03-09T00:04:19.284763+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:04:19.284763+0000","last_undegraded":"2024-03-09T00:04:19.284763+0000","last_fullsized":"2024-03-09T00:04:19.284763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937662+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:04:18.937662+0000","last_peered":"2024-03-09T00:04:18.937662+0000","last_clean":"2024-03-09T00:04:18.937662+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:04:18.937662+0000","last_undegraded":"2024-03-09T00:04:18.937662+0000","last_fullsized":"2024-03-09T00:04:18.937662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940234+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:04:18.940234+0000","last_peered":"2024-03-09T00:04:18.940234+0000","last_clean":"2024-03-09T00:04:18.940234+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:04:18.940234+0000","last_undegraded":"2024-03-09T00:04:18.940234+0000","last_fullsized":"2024-03-09T00:04:18.940234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940166+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:04:18.940166+0000","last_peered":"2024-03-09T00:04:18.940166+0000","last_clean":"2024-03-09T00:04:18.940166+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:04:18.940166+0000","last_undegraded":"2024-03-09T00:04:18.940166+0000","last_fullsized":"2024-03-09T00:04:18.940166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284281+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:04:19.284281+0000","last_peered":"2024-03-09T00:04:19.284281+0000","last_clean":"2024-03-09T00:04:19.284281+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:04:19.284281+0000","last_undegraded":"2024-03-09T00:04:19.284281+0000","last_fullsized":"2024-03-09T00:04:19.284281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285344+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:04:19.285344+0000","last_peered":"2024-03-09T00:04:19.285344+0000","last_clean":"2024-03-09T00:04:19.285344+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:04:19.285344+0000","last_undegraded":"2024-03-09T00:04:19.285344+0000","last_fullsized":"2024-03-09T00:04:19.285344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284460+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:04:19.284460+0000","last_peered":"2024-03-09T00:04:19.284460+0000","last_clean":"2024-03-09T00:04:19.284460+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:04:19.284460+0000","last_undegraded":"2024-03-09T00:04:19.284460+0000","last_fullsized":"2024-03-09T00:04:19.284460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286945+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:04:19.286945+0000","last_peered":"2024-03-09T00:04:19.286945+0000","last_clean":"2024-03-09T00:04:19.286945+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:04:19.286945+0000","last_undegraded":"2024-03-09T00:04:19.286945+0000","last_fullsized":"2024-03-09T00:04:19.286945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286254+0000","last_change":"2024-03-09T00:04:17.955496+0000","last_active":"2024-03-09T00:04:19.286254+0000","last_peered":"2024-03-09T00:04:19.286254+0000","last_clean":"2024-03-09T00:04:19.286254+0000","last_became_active":"2024-03-09T00:04:17.954710+0000","last_became_peered":"2024-03-09T00:04:17.954710+0000","last_unstale":"2024-03-09T00:04:19.286254+0000","last_undegraded":"2024-03-09T00:04:19.286254+0000","last_fullsized":"2024-03-09T00:04:19.286254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:35:05.318796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284589+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:04:19.284589+0000","last_peered":"2024-03-09T00:04:19.284589+0000","last_clean":"2024-03-09T00:04:19.284589+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:04:19.284589+0000","last_undegraded":"2024-03-09T00:04:19.284589+0000","last_fullsized":"2024-03-09T00:04:19.284589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285199+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:04:19.285199+0000","last_peered":"2024-03-09T00:04:19.285199+0000","last_clean":"2024-03-09T00:04:19.285199+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:04:19.285199+0000","last_undegraded":"2024-03-09T00:04:19.285199+0000","last_fullsized":"2024-03-09T00:04:19.285199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284377+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:04:19.284377+0000","last_peered":"2024-03-09T00:04:19.284377+0000","last_clean":"2024-03-09T00:04:19.284377+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:04:19.284377+0000","last_undegraded":"2024-03-09T00:04:19.284377+0000","last_fullsized":"2024-03-09T00:04:19.284377+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285322+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:04:19.285322+0000","last_peered":"2024-03-09T00:04:19.285322+0000","last_clean":"2024-03-09T00:04:19.285322+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:04:19.285322+0000","last_undegraded":"2024-03-09T00:04:19.285322+0000","last_fullsized":"2024-03-09T00:04:19.285322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940765+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:04:18.940765+0000","last_peered":"2024-03-09T00:04:18.940765+0000","last_clean":"2024-03-09T00:04:18.940765+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:04:18.940765+0000","last_undegraded":"2024-03-09T00:04:18.940765+0000","last_fullsized":"2024-03-09T00:04:18.940765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284721+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:04:19.284721+0000","last_peered":"2024-03-09T00:04:19.284721+0000","last_clean":"2024-03-09T00:04:19.284721+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:04:19.284721+0000","last_undegraded":"2024-03-09T00:04:19.284721+0000","last_fullsized":"2024-03-09T00:04:19.284721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285434+0000","last_change":"2024-03-09T00:04:13.334422+0000","last_active":"2024-03-09T00:04:19.285434+0000","last_peered":"2024-03-09T00:04:19.285434+0000","last_clean":"2024-03-09T00:04:19.285434+0000","last_became_active":"2024-03-09T00:04:13.334101+0000","last_became_peered":"2024-03-09T00:04:13.334101+0000","last_unstale":"2024-03-09T00:04:19.285434+0000","last_undegraded":"2024-03-09T00:04:19.285434+0000","last_fullsized":"2024-03-09T00:04:19.285434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:04:08.860700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285247+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:04:19.285247+0000","last_peered":"2024-03-09T00:04:19.285247+0000","last_clean":"2024-03-09T00:04:19.285247+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:04:19.285247+0000","last_undegraded":"2024-03-09T00:04:19.285247+0000","last_fullsized":"2024-03-09T00:04:19.285247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284658+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:04:19.284658+0000","last_peered":"2024-03-09T00:04:19.284658+0000","last_clean":"2024-03-09T00:04:19.284658+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:04:19.284658+0000","last_undegraded":"2024-03-09T00:04:19.284658+0000","last_fullsized":"2024-03-09T00:04:19.284658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284812+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:04:19.284812+0000","last_peered":"2024-03-09T00:04:19.284812+0000","last_clean":"2024-03-09T00:04:19.284812+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:04:19.284812+0000","last_undegraded":"2024-03-09T00:04:19.284812+0000","last_fullsized":"2024-03-09T00:04:19.284812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286016+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:04:19.286016+0000","last_peered":"2024-03-09T00:04:19.286016+0000","last_clean":"2024-03-09T00:04:19.286016+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:04:19.286016+0000","last_undegraded":"2024-03-09T00:04:19.286016+0000","last_fullsized":"2024-03-09T00:04:19.286016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286922+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:04:19.286922+0000","last_peered":"2024-03-09T00:04:19.286922+0000","last_clean":"2024-03-09T00:04:19.286922+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:04:19.286922+0000","last_undegraded":"2024-03-09T00:04:19.286922+0000","last_fullsized":"2024-03-09T00:04:19.286922+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285818+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:04:19.285818+0000","last_peered":"2024-03-09T00:04:19.285818+0000","last_clean":"2024-03-09T00:04:19.285818+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:04:19.285818+0000","last_undegraded":"2024-03-09T00:04:19.285818+0000","last_fullsized":"2024-03-09T00:04:19.285818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284969+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:04:19.284969+0000","last_peered":"2024-03-09T00:04:19.284969+0000","last_clean":"2024-03-09T00:04:19.284969+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:04:19.284969+0000","last_undegraded":"2024-03-09T00:04:19.284969+0000","last_fullsized":"2024-03-09T00:04:19.284969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286972+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:04:19.286972+0000","last_peered":"2024-03-09T00:04:19.286972+0000","last_clean":"2024-03-09T00:04:19.286972+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:04:19.286972+0000","last_undegraded":"2024-03-09T00:04:19.286972+0000","last_fullsized":"2024-03-09T00:04:19.286972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938336+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:04:18.938336+0000","last_peered":"2024-03-09T00:04:18.938336+0000","last_clean":"2024-03-09T00:04:18.938336+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:04:18.938336+0000","last_undegraded":"2024-03-09T00:04:18.938336+0000","last_fullsized":"2024-03-09T00:04:18.938336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285928+0000","last_change":"2024-03-09T00:04:13.935057+0000","last_active":"2024-03-09T00:04:19.285928+0000","last_peered":"2024-03-09T00:04:19.285928+0000","last_clean":"2024-03-09T00:04:19.285928+0000","last_became_active":"2024-03-09T00:04:13.934719+0000","last_became_peered":"2024-03-09T00:04:13.934719+0000","last_unstale":"2024-03-09T00:04:19.285928+0000","last_undegraded":"2024-03-09T00:04:19.285928+0000","last_fullsized":"2024-03-09T00:04:19.285928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:24:41.705525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285278+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:04:19.285278+0000","last_peered":"2024-03-09T00:04:19.285278+0000","last_clean":"2024-03-09T00:04:19.285278+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:04:19.285278+0000","last_undegraded":"2024-03-09T00:04:19.285278+0000","last_fullsized":"2024-03-09T00:04:19.285278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937399+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:04:18.937399+0000","last_peered":"2024-03-09T00:04:18.937399+0000","last_clean":"2024-03-09T00:04:18.937399+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:04:18.937399+0000","last_undegraded":"2024-03-09T00:04:18.937399+0000","last_fullsized":"2024-03-09T00:04:18.937399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284153+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:04:19.284153+0000","last_peered":"2024-03-09T00:04:19.284153+0000","last_clean":"2024-03-09T00:04:19.284153+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:04:19.284153+0000","last_undegraded":"2024-03-09T00:04:19.284153+0000","last_fullsized":"2024-03-09T00:04:19.284153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940885+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:04:18.940885+0000","last_peered":"2024-03-09T00:04:18.940885+0000","last_clean":"2024-03-09T00:04:18.940885+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:04:18.940885+0000","last_undegraded":"2024-03-09T00:04:18.940885+0000","last_fullsized":"2024-03-09T00:04:18.940885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284843+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:04:19.284843+0000","last_peered":"2024-03-09T00:04:19.284843+0000","last_clean":"2024-03-09T00:04:19.284843+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:04:19.284843+0000","last_undegraded":"2024-03-09T00:04:19.284843+0000","last_fullsized":"2024-03-09T00:04:19.284843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284037+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:04:19.284037+0000","last_peered":"2024-03-09T00:04:19.284037+0000","last_clean":"2024-03-09T00:04:19.284037+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:04:19.284037+0000","last_undegraded":"2024-03-09T00:04:19.284037+0000","last_fullsized":"2024-03-09T00:04:19.284037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284237+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:04:19.284237+0000","last_peered":"2024-03-09T00:04:19.284237+0000","last_clean":"2024-03-09T00:04:19.284237+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:04:19.284237+0000","last_undegraded":"2024-03-09T00:04:19.284237+0000","last_fullsized":"2024-03-09T00:04:19.284237+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938748+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:04:18.938748+0000","last_peered":"2024-03-09T00:04:18.938748+0000","last_clean":"2024-03-09T00:04:18.938748+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:04:18.938748+0000","last_undegraded":"2024-03-09T00:04:18.938748+0000","last_fullsized":"2024-03-09T00:04:18.938748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285004+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:04:19.285004+0000","last_peered":"2024-03-09T00:04:19.285004+0000","last_clean":"2024-03-09T00:04:19.285004+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:04:19.285004+0000","last_undegraded":"2024-03-09T00:04:19.285004+0000","last_fullsized":"2024-03-09T00:04:19.285004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.283875+0000","last_change":"2024-03-09T00:04:17.960827+0000","last_active":"2024-03-09T00:04:19.283875+0000","last_peered":"2024-03-09T00:04:19.283875+0000","last_clean":"2024-03-09T00:04:19.283875+0000","last_became_active":"2024-03-09T00:04:17.959916+0000","last_became_peered":"2024-03-09T00:04:17.959916+0000","last_unstale":"2024-03-09T00:04:19.283875+0000","last_undegraded":"2024-03-09T00:04:19.283875+0000","last_fullsized":"2024-03-09T00:04:19.283875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:23:04.325826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.334152+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.333835+0000","last_became_peered":"2024-03-09T00:04:13.333835+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:19:49.203289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940768+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:04:18.940768+0000","last_peered":"2024-03-09T00:04:18.940768+0000","last_clean":"2024-03-09T00:04:18.940768+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:04:18.940768+0000","last_undegraded":"2024-03-09T00:04:18.940768+0000","last_fullsized":"2024-03-09T00:04:18.940768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285041+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:04:19.285041+0000","last_peered":"2024-03-09T00:04:19.285041+0000","last_clean":"2024-03-09T00:04:19.285041+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:04:19.285041+0000","last_undegraded":"2024-03-09T00:04:19.285041+0000","last_fullsized":"2024-03-09T00:04:19.285041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285903+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:04:19.285903+0000","last_peered":"2024-03-09T00:04:19.285903+0000","last_clean":"2024-03-09T00:04:19.285903+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:04:19.285903+0000","last_undegraded":"2024-03-09T00:04:19.285903+0000","last_fullsized":"2024-03-09T00:04:19.285903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285350+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:04:19.285350+0000","last_peered":"2024-03-09T00:04:19.285350+0000","last_clean":"2024-03-09T00:04:19.285350+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:04:19.285350+0000","last_undegraded":"2024-03-09T00:04:19.285350+0000","last_fullsized":"2024-03-09T00:04:19.285350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284040+0000","last_change":"2024-03-09T00:04:17.958243+0000","last_active":"2024-03-09T00:04:19.284040+0000","last_peered":"2024-03-09T00:04:19.284040+0000","last_clean":"2024-03-09T00:04:19.284040+0000","last_became_active":"2024-03-09T00:04:17.957825+0000","last_became_peered":"2024-03-09T00:04:17.957825+0000","last_unstale":"2024-03-09T00:04:19.284040+0000","last_undegraded":"2024-03-09T00:04:19.284040+0000","last_fullsized":"2024-03-09T00:04:19.284040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:12.927363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:38.992812+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:04:38.992812+0000","last_peered":"2024-03-09T00:04:38.992812+0000","last_clean":"2024-03-09T00:04:38.992812+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:04:38.992812+0000","last_undegraded":"2024-03-09T00:04:38.992812+0000","last_fullsized":"2024-03-09T00:04:38.992812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285266+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:04:19.285266+0000","last_peered":"2024-03-09T00:04:19.285266+0000","last_clean":"2024-03-09T00:04:19.285266+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:04:19.285266+0000","last_undegraded":"2024-03-09T00:04:19.285266+0000","last_fullsized":"2024-03-09T00:04:19.285266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286420+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:04:19.286420+0000","last_peered":"2024-03-09T00:04:19.286420+0000","last_clean":"2024-03-09T00:04:19.286420+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:04:19.286420+0000","last_undegraded":"2024-03-09T00:04:19.286420+0000","last_fullsized":"2024-03-09T00:04:19.286420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286747+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:04:19.286747+0000","last_peered":"2024-03-09T00:04:19.286747+0000","last_clean":"2024-03-09T00:04:19.286747+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:04:19.286747+0000","last_undegraded":"2024-03-09T00:04:19.286747+0000","last_fullsized":"2024-03-09T00:04:19.286747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937495+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:04:18.937495+0000","last_peered":"2024-03-09T00:04:18.937495+0000","last_clean":"2024-03-09T00:04:18.937495+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:04:18.937495+0000","last_undegraded":"2024-03-09T00:04:18.937495+0000","last_fullsized":"2024-03-09T00:04:18.937495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285804+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:04:19.285804+0000","last_peered":"2024-03-09T00:04:19.285804+0000","last_clean":"2024-03-09T00:04:19.285804+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:04:19.285804+0000","last_undegraded":"2024-03-09T00:04:19.285804+0000","last_fullsized":"2024-03-09T00:04:19.285804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286331+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:04:19.286331+0000","last_peered":"2024-03-09T00:04:19.286331+0000","last_clean":"2024-03-09T00:04:19.286331+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:04:19.286331+0000","last_undegraded":"2024-03-09T00:04:19.286331+0000","last_fullsized":"2024-03-09T00:04:19.286331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116312+0000","last_change":"2024-03-09T00:04:13.935230+0000","last_active":"2024-03-09T00:04:19.116312+0000","last_peered":"2024-03-09T00:04:19.116312+0000","last_clean":"2024-03-09T00:04:19.116312+0000","last_became_active":"2024-03-09T00:04:13.934838+0000","last_became_peered":"2024-03-09T00:04:13.934838+0000","last_unstale":"2024-03-09T00:04:19.116312+0000","last_undegraded":"2024-03-09T00:04:19.116312+0000","last_fullsized":"2024-03-09T00:04:19.116312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:37.002289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284230+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:04:19.284230+0000","last_peered":"2024-03-09T00:04:19.284230+0000","last_clean":"2024-03-09T00:04:19.284230+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:04:19.284230+0000","last_undegraded":"2024-03-09T00:04:19.284230+0000","last_fullsized":"2024-03-09T00:04:19.284230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284338+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:04:19.284338+0000","last_peered":"2024-03-09T00:04:19.284338+0000","last_clean":"2024-03-09T00:04:19.284338+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:04:19.284338+0000","last_undegraded":"2024-03-09T00:04:19.284338+0000","last_fullsized":"2024-03-09T00:04:19.284338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.115942+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:04:19.115942+0000","last_peered":"2024-03-09T00:04:19.115942+0000","last_clean":"2024-03-09T00:04:19.115942+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:04:19.115942+0000","last_undegraded":"2024-03-09T00:04:19.115942+0000","last_fullsized":"2024-03-09T00:04:19.115942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937983+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:04:18.937983+0000","last_peered":"2024-03-09T00:04:18.937983+0000","last_clean":"2024-03-09T00:04:18.937983+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:04:18.937983+0000","last_undegraded":"2024-03-09T00:04:18.937983+0000","last_fullsized":"2024-03-09T00:04:18.937983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284633+0000","last_change":"2024-03-09T00:04:13.330355+0000","last_active":"2024-03-09T00:04:19.284633+0000","last_peered":"2024-03-09T00:04:19.284633+0000","last_clean":"2024-03-09T00:04:19.284633+0000","last_became_active":"2024-03-09T00:04:13.330012+0000","last_became_peered":"2024-03-09T00:04:13.330012+0000","last_unstale":"2024-03-09T00:04:19.284633+0000","last_undegraded":"2024-03-09T00:04:19.284633+0000","last_fullsized":"2024-03-09T00:04:19.284633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:46.699807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285690+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:04:19.285690+0000","last_peered":"2024-03-09T00:04:19.285690+0000","last_clean":"2024-03-09T00:04:19.285690+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:04:19.285690+0000","last_undegraded":"2024-03-09T00:04:19.285690+0000","last_fullsized":"2024-03-09T00:04:19.285690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285653+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:04:19.285653+0000","last_peered":"2024-03-09T00:04:19.285653+0000","last_clean":"2024-03-09T00:04:19.285653+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:04:19.285653+0000","last_undegraded":"2024-03-09T00:04:19.285653+0000","last_fullsized":"2024-03-09T00:04:19.285653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285009+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:04:19.285009+0000","last_peered":"2024-03-09T00:04:19.285009+0000","last_clean":"2024-03-09T00:04:19.285009+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:04:19.285009+0000","last_undegraded":"2024-03-09T00:04:19.285009+0000","last_fullsized":"2024-03-09T00:04:19.285009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284962+0000","last_change":"2024-03-09T00:04:13.934570+0000","last_active":"2024-03-09T00:04:19.284962+0000","last_peered":"2024-03-09T00:04:19.284962+0000","last_clean":"2024-03-09T00:04:19.284962+0000","last_became_active":"2024-03-09T00:04:13.934219+0000","last_became_peered":"2024-03-09T00:04:13.934219+0000","last_unstale":"2024-03-09T00:04:19.284962+0000","last_undegraded":"2024-03-09T00:04:19.284962+0000","last_fullsized":"2024-03-09T00:04:19.284962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:37:39.694115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285000+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:04:19.285000+0000","last_peered":"2024-03-09T00:04:19.285000+0000","last_clean":"2024-03-09T00:04:19.285000+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:04:19.285000+0000","last_undegraded":"2024-03-09T00:04:19.285000+0000","last_fullsized":"2024-03-09T00:04:19.285000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284300+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:04:19.284300+0000","last_peered":"2024-03-09T00:04:19.284300+0000","last_clean":"2024-03-09T00:04:19.284300+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:04:19.284300+0000","last_undegraded":"2024-03-09T00:04:19.284300+0000","last_fullsized":"2024-03-09T00:04:19.284300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938530+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:04:18.938530+0000","last_peered":"2024-03-09T00:04:18.938530+0000","last_clean":"2024-03-09T00:04:18.938530+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:04:18.938530+0000","last_undegraded":"2024-03-09T00:04:18.938530+0000","last_fullsized":"2024-03-09T00:04:18.938530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937720+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:04:18.937720+0000","last_peered":"2024-03-09T00:04:18.937720+0000","last_clean":"2024-03-09T00:04:18.937720+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:04:18.937720+0000","last_undegraded":"2024-03-09T00:04:18.937720+0000","last_fullsized":"2024-03-09T00:04:18.937720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938344+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:04:18.938344+0000","last_peered":"2024-03-09T00:04:18.938344+0000","last_clean":"2024-03-09T00:04:18.938344+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:04:18.938344+0000","last_undegraded":"2024-03-09T00:04:18.938344+0000","last_fullsized":"2024-03-09T00:04:18.938344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937783+0000","last_change":"2024-03-09T00:04:18.031380+0000","last_active":"2024-03-09T00:04:18.937783+0000","last_peered":"2024-03-09T00:04:18.937783+0000","last_clean":"2024-03-09T00:04:18.937783+0000","last_became_active":"2024-03-09T00:04:18.030976+0000","last_became_peered":"2024-03-09T00:04:18.030976+0000","last_unstale":"2024-03-09T00:04:18.937783+0000","last_undegraded":"2024-03-09T00:04:18.937783+0000","last_fullsized":"2024-03-09T00:04:18.937783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:25:18.264867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285050+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:04:19.285050+0000","last_peered":"2024-03-09T00:04:19.285050+0000","last_clean":"2024-03-09T00:04:19.285050+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:04:19.285050+0000","last_undegraded":"2024-03-09T00:04:19.285050+0000","last_fullsized":"2024-03-09T00:04:19.285050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937260+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:04:18.937260+0000","last_peered":"2024-03-09T00:04:18.937260+0000","last_clean":"2024-03-09T00:04:18.937260+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:04:18.937260+0000","last_undegraded":"2024-03-09T00:04:18.937260+0000","last_fullsized":"2024-03-09T00:04:18.937260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:21.234408+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:04:21.234408+0000","last_peered":"2024-03-09T00:04:21.234408+0000","last_clean":"2024-03-09T00:04:21.234408+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:04:21.234408+0000","last_undegraded":"2024-03-09T00:04:21.234408+0000","last_fullsized":"2024-03-09T00:04:21.234408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284449+0000","last_change":"2024-03-09T00:04:13.939193+0000","last_active":"2024-03-09T00:04:19.284449+0000","last_peered":"2024-03-09T00:04:19.284449+0000","last_clean":"2024-03-09T00:04:19.284449+0000","last_became_active":"2024-03-09T00:04:13.938679+0000","last_became_peered":"2024-03-09T00:04:13.938679+0000","last_unstale":"2024-03-09T00:04:19.284449+0000","last_undegraded":"2024-03-09T00:04:19.284449+0000","last_fullsized":"2024-03-09T00:04:19.284449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:57:40.949182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284794+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:04:19.284794+0000","last_peered":"2024-03-09T00:04:19.284794+0000","last_clean":"2024-03-09T00:04:19.284794+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:04:19.284794+0000","last_undegraded":"2024-03-09T00:04:19.284794+0000","last_fullsized":"2024-03-09T00:04:19.284794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.115937+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:04:19.115937+0000","last_peered":"2024-03-09T00:04:19.115937+0000","last_clean":"2024-03-09T00:04:19.115937+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:04:19.115937+0000","last_undegraded":"2024-03-09T00:04:19.115937+0000","last_fullsized":"2024-03-09T00:04:19.115937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285678+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:04:19.285678+0000","last_peered":"2024-03-09T00:04:19.285678+0000","last_clean":"2024-03-09T00:04:19.285678+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:04:19.285678+0000","last_undegraded":"2024-03-09T00:04:19.285678+0000","last_fullsized":"2024-03-09T00:04:19.285678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937256+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:04:18.937256+0000","last_peered":"2024-03-09T00:04:18.937256+0000","last_clean":"2024-03-09T00:04:18.937256+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:04:18.937256+0000","last_undegraded":"2024-03-09T00:04:18.937256+0000","last_fullsized":"2024-03-09T00:04:18.937256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116201+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:04:19.116201+0000","last_peered":"2024-03-09T00:04:19.116201+0000","last_clean":"2024-03-09T00:04:19.116201+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:04:19.116201+0000","last_undegraded":"2024-03-09T00:04:19.116201+0000","last_fullsized":"2024-03-09T00:04:19.116201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284788+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:04:19.284788+0000","last_peered":"2024-03-09T00:04:19.284788+0000","last_clean":"2024-03-09T00:04:19.284788+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:04:19.284788+0000","last_undegraded":"2024-03-09T00:04:19.284788+0000","last_fullsized":"2024-03-09T00:04:19.284788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285365+0000","last_change":"2024-03-09T00:04:13.331922+0000","last_active":"2024-03-09T00:04:19.285365+0000","last_peered":"2024-03-09T00:04:19.285365+0000","last_clean":"2024-03-09T00:04:19.285365+0000","last_became_active":"2024-03-09T00:04:13.331610+0000","last_became_peered":"2024-03-09T00:04:13.331610+0000","last_unstale":"2024-03-09T00:04:19.285365+0000","last_undegraded":"2024-03-09T00:04:19.285365+0000","last_fullsized":"2024-03-09T00:04:19.285365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:29.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937868+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:04:18.937868+0000","last_peered":"2024-03-09T00:04:18.937868+0000","last_clean":"2024-03-09T00:04:18.937868+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:04:18.937868+0000","last_undegraded":"2024-03-09T00:04:18.937868+0000","last_fullsized":"2024-03-09T00:04:18.937868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284102+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:04:19.284102+0000","last_peered":"2024-03-09T00:04:19.284102+0000","last_clean":"2024-03-09T00:04:19.284102+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:04:19.284102+0000","last_undegraded":"2024-03-09T00:04:19.284102+0000","last_fullsized":"2024-03-09T00:04:19.284102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284558+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:04:19.284558+0000","last_peered":"2024-03-09T00:04:19.284558+0000","last_clean":"2024-03-09T00:04:19.284558+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:04:19.284558+0000","last_undegraded":"2024-03-09T00:04:19.284558+0000","last_fullsized":"2024-03-09T00:04:19.284558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285217+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:04:19.285217+0000","last_peered":"2024-03-09T00:04:19.285217+0000","last_clean":"2024-03-09T00:04:19.285217+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:04:19.285217+0000","last_undegraded":"2024-03-09T00:04:19.285217+0000","last_fullsized":"2024-03-09T00:04:19.285217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286701+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:04:19.286701+0000","last_peered":"2024-03-09T00:04:19.286701+0000","last_clean":"2024-03-09T00:04:19.286701+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:04:19.286701+0000","last_undegraded":"2024-03-09T00:04:19.286701+0000","last_fullsized":"2024-03-09T00:04:19.286701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285219+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:04:19.285219+0000","last_peered":"2024-03-09T00:04:19.285219+0000","last_clean":"2024-03-09T00:04:19.285219+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:04:19.285219+0000","last_undegraded":"2024-03-09T00:04:19.285219+0000","last_fullsized":"2024-03-09T00:04:19.285219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940951+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:04:18.940951+0000","last_peered":"2024-03-09T00:04:18.940951+0000","last_clean":"2024-03-09T00:04:18.940951+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:04:18.940951+0000","last_undegraded":"2024-03-09T00:04:18.940951+0000","last_fullsized":"2024-03-09T00:04:18.940951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284781+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:04:19.284781+0000","last_peered":"2024-03-09T00:04:19.284781+0000","last_clean":"2024-03-09T00:04:19.284781+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:04:19.284781+0000","last_undegraded":"2024-03-09T00:04:19.284781+0000","last_fullsized":"2024-03-09T00:04:19.284781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284405+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:04:19.284405+0000","last_peered":"2024-03-09T00:04:19.284405+0000","last_clean":"2024-03-09T00:04:19.284405+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:04:19.284405+0000","last_undegraded":"2024-03-09T00:04:19.284405+0000","last_fullsized":"2024-03-09T00:04:19.284405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284531+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:04:19.284531+0000","last_peered":"2024-03-09T00:04:19.284531+0000","last_clean":"2024-03-09T00:04:19.284531+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:04:19.284531+0000","last_undegraded":"2024-03-09T00:04:19.284531+0000","last_fullsized":"2024-03-09T00:04:19.284531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284485+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:04:19.284485+0000","last_peered":"2024-03-09T00:04:19.284485+0000","last_clean":"2024-03-09T00:04:19.284485+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:04:19.284485+0000","last_undegraded":"2024-03-09T00:04:19.284485+0000","last_fullsized":"2024-03-09T00:04:19.284485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284303+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:04:19.284303+0000","last_peered":"2024-03-09T00:04:19.284303+0000","last_clean":"2024-03-09T00:04:19.284303+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:04:19.284303+0000","last_undegraded":"2024-03-09T00:04:19.284303+0000","last_fullsized":"2024-03-09T00:04:19.284303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284537+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:04:19.284537+0000","last_peered":"2024-03-09T00:04:19.284537+0000","last_clean":"2024-03-09T00:04:19.284537+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:04:19.284537+0000","last_undegraded":"2024-03-09T00:04:19.284537+0000","last_fullsized":"2024-03-09T00:04:19.284537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285237+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:04:19.285237+0000","last_peered":"2024-03-09T00:04:19.285237+0000","last_clean":"2024-03-09T00:04:19.285237+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:04:19.285237+0000","last_undegraded":"2024-03-09T00:04:19.285237+0000","last_fullsized":"2024-03-09T00:04:19.285237+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285989+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:04:19.285989+0000","last_peered":"2024-03-09T00:04:19.285989+0000","last_clean":"2024-03-09T00:04:19.285989+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:04:19.285989+0000","last_undegraded":"2024-03-09T00:04:19.285989+0000","last_fullsized":"2024-03-09T00:04:19.285989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284711+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:04:19.284711+0000","last_peered":"2024-03-09T00:04:19.284711+0000","last_clean":"2024-03-09T00:04:19.284711+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:04:19.284711+0000","last_undegraded":"2024-03-09T00:04:19.284711+0000","last_fullsized":"2024-03-09T00:04:19.284711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285355+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:04:19.285355+0000","last_peered":"2024-03-09T00:04:19.285355+0000","last_clean":"2024-03-09T00:04:19.285355+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:04:19.285355+0000","last_undegraded":"2024-03-09T00:04:19.285355+0000","last_fullsized":"2024-03-09T00:04:19.285355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284811+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:04:19.284811+0000","last_peered":"2024-03-09T00:04:19.284811+0000","last_clean":"2024-03-09T00:04:19.284811+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:04:19.284811+0000","last_undegraded":"2024-03-09T00:04:19.284811+0000","last_fullsized":"2024-03-09T00:04:19.284811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116084+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:04:19.116084+0000","last_peered":"2024-03-09T00:04:19.116084+0000","last_clean":"2024-03-09T00:04:19.116084+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:04:19.116084+0000","last_undegraded":"2024-03-09T00:04:19.116084+0000","last_fullsized":"2024-03-09T00:04:19.116084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284635+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:04:19.284635+0000","last_peered":"2024-03-09T00:04:19.284635+0000","last_clean":"2024-03-09T00:04:19.284635+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:04:19.284635+0000","last_undegraded":"2024-03-09T00:04:19.284635+0000","last_fullsized":"2024-03-09T00:04:19.284635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.287034+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:04:19.287034+0000","last_peered":"2024-03-09T00:04:19.287034+0000","last_clean":"2024-03-09T00:04:19.287034+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:04:19.287034+0000","last_undegraded":"2024-03-09T00:04:19.287034+0000","last_fullsized":"2024-03-09T00:04:19.287034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116620+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:04:19.116620+0000","last_peered":"2024-03-09T00:04:19.116620+0000","last_clean":"2024-03-09T00:04:19.116620+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:04:19.116620+0000","last_undegraded":"2024-03-09T00:04:19.116620+0000","last_fullsized":"2024-03-09T00:04:19.116620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940290+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:04:18.940290+0000","last_peered":"2024-03-09T00:04:18.940290+0000","last_clean":"2024-03-09T00:04:18.940290+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:04:18.940290+0000","last_undegraded":"2024-03-09T00:04:18.940290+0000","last_fullsized":"2024-03-09T00:04:18.940290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286176+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:04:19.286176+0000","last_peered":"2024-03-09T00:04:19.286176+0000","last_clean":"2024-03-09T00:04:19.286176+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:04:19.286176+0000","last_undegraded":"2024-03-09T00:04:19.286176+0000","last_fullsized":"2024-03-09T00:04:19.286176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285655+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:04:19.285655+0000","last_peered":"2024-03-09T00:04:19.285655+0000","last_clean":"2024-03-09T00:04:19.285655+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:04:19.285655+0000","last_undegraded":"2024-03-09T00:04:19.285655+0000","last_fullsized":"2024-03-09T00:04:19.285655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937927+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:04:18.937927+0000","last_peered":"2024-03-09T00:04:18.937927+0000","last_clean":"2024-03-09T00:04:18.937927+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:04:18.937927+0000","last_undegraded":"2024-03-09T00:04:18.937927+0000","last_fullsized":"2024-03-09T00:04:18.937927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116560+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:04:19.116560+0000","last_peered":"2024-03-09T00:04:19.116560+0000","last_clean":"2024-03-09T00:04:19.116560+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:04:19.116560+0000","last_undegraded":"2024-03-09T00:04:19.116560+0000","last_fullsized":"2024-03-09T00:04:19.116560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284687+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:04:19.284687+0000","last_peered":"2024-03-09T00:04:19.284687+0000","last_clean":"2024-03-09T00:04:19.284687+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:04:19.284687+0000","last_undegraded":"2024-03-09T00:04:19.284687+0000","last_fullsized":"2024-03-09T00:04:19.284687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286441+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:04:19.286441+0000","last_peered":"2024-03-09T00:04:19.286441+0000","last_clean":"2024-03-09T00:04:19.286441+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:04:19.286441+0000","last_undegraded":"2024-03-09T00:04:19.286441+0000","last_fullsized":"2024-03-09T00:04:19.286441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285867+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:04:19.285867+0000","last_peered":"2024-03-09T00:04:19.285867+0000","last_clean":"2024-03-09T00:04:19.285867+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:04:19.285867+0000","last_undegraded":"2024-03-09T00:04:19.285867+0000","last_fullsized":"2024-03-09T00:04:19.285867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940559+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:04:18.940559+0000","last_peered":"2024-03-09T00:04:18.940559+0000","last_clean":"2024-03-09T00:04:18.940559+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:04:18.940559+0000","last_undegraded":"2024-03-09T00:04:18.940559+0000","last_fullsized":"2024-03-09T00:04:18.940559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937860+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:04:18.937860+0000","last_peered":"2024-03-09T00:04:18.937860+0000","last_clean":"2024-03-09T00:04:18.937860+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:04:18.937860+0000","last_undegraded":"2024-03-09T00:04:18.937860+0000","last_fullsized":"2024-03-09T00:04:18.937860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285258+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:04:19.285258+0000","last_peered":"2024-03-09T00:04:19.285258+0000","last_clean":"2024-03-09T00:04:19.285258+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:04:19.285258+0000","last_undegraded":"2024-03-09T00:04:19.285258+0000","last_fullsized":"2024-03-09T00:04:19.285258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286463+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:04:19.286463+0000","last_peered":"2024-03-09T00:04:19.286463+0000","last_clean":"2024-03-09T00:04:19.286463+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:04:19.286463+0000","last_undegraded":"2024-03-09T00:04:19.286463+0000","last_fullsized":"2024-03-09T00:04:19.286463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285311+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:04:19.285311+0000","last_peered":"2024-03-09T00:04:19.285311+0000","last_clean":"2024-03-09T00:04:19.285311+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:04:19.285311+0000","last_undegraded":"2024-03-09T00:04:19.285311+0000","last_fullsized":"2024-03-09T00:04:19.285311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284953+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:04:19.284953+0000","last_peered":"2024-03-09T00:04:19.284953+0000","last_clean":"2024-03-09T00:04:19.284953+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:04:19.284953+0000","last_undegraded":"2024-03-09T00:04:19.284953+0000","last_fullsized":"2024-03-09T00:04:19.284953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284416+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:04:19.284416+0000","last_peered":"2024-03-09T00:04:19.284416+0000","last_clean":"2024-03-09T00:04:19.284416+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:04:19.284416+0000","last_undegraded":"2024-03-09T00:04:19.284416+0000","last_fullsized":"2024-03-09T00:04:19.284416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284981+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:04:19.284981+0000","last_peered":"2024-03-09T00:04:19.284981+0000","last_clean":"2024-03-09T00:04:19.284981+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:04:19.284981+0000","last_undegraded":"2024-03-09T00:04:19.284981+0000","last_fullsized":"2024-03-09T00:04:19.284981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937801+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:04:18.937801+0000","last_peered":"2024-03-09T00:04:18.937801+0000","last_clean":"2024-03-09T00:04:18.937801+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:04:18.937801+0000","last_undegraded":"2024-03-09T00:04:18.937801+0000","last_fullsized":"2024-03-09T00:04:18.937801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285884+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:04:19.285884+0000","last_peered":"2024-03-09T00:04:19.285884+0000","last_clean":"2024-03-09T00:04:19.285884+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:04:19.285884+0000","last_undegraded":"2024-03-09T00:04:19.285884+0000","last_fullsized":"2024-03-09T00:04:19.285884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":20520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32740,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712412,"statfs":{"total":95995035648,"available":95961509888,"internally_reserved":0,"allocated":1601536,"data_stored":1071619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712424,"statfs":{"total":95995035648,"available":95961522176,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36896,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708256,"statfs":{"total":95995035648,"available":95957254144,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":10476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:04:42.683 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 -- 172.21.15.73:0/2100678115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 msgr2=0x7fabdc100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 --2- 172.21.15.73:0/2100678115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc100630 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fabd4009ca0 tx=0x7fabd402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 -- 172.21.15.73:0/2100678115 shutdown_connections 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 --2- 172.21.15.73:0/2100678115 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 0x7fabdc108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 --2- 172.21.15.73:0/2100678115 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 --2- 172.21.15.73:0/2100678115 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc100630 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 -- 172.21.15.73:0/2100678115 >> 172.21.15.73:0/2100678115 conn(0x7fabdc0fb780 msgr2=0x7fabdc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 -- 172.21.15.73:0/2100678115 shutdown_connections 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.825+0000 7fabe3169640 1 -- 172.21.15.73:0/2100678115 wait complete. 2024-03-09T00:04:42.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 Processor -- start 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 -- start start 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 0x7fabdc19ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc10bf00 con 0x7fabdc100210 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc10bd80 con 0x7fabdc101400 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe3169640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabdc10c080 con 0x7fabdc101dc0 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35638/0 (socket says 172.21.15.73:35638) 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 -- 172.21.15.73:0/2961375941 learned_addr learned my addr 172.21.15.73:0/2961375941 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe1706640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 0x7fabdc19ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabdbfff640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 -- 172.21.15.73:0/2961375941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 msgr2=0x7fabdc19ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 0x7fabdc19ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 -- 172.21.15.73:0/2961375941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 msgr2=0x7fabdc197be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc197be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabe0f05640 1 -- 172.21.15.73:0/2961375941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabdc19f2c0 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.826+0000 7fabdbfff640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabe0f05640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fabd40324c0 tx=0x7fabd4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd4039040 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fabd4044070 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fabd40042a0 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabdc19f550 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.827+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabdc19fa40 con 0x7fabdc100210 2024-03-09T00:04:42.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.828+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabdc19fca0 con 0x7fabdc100210 2024-03-09T00:04:42.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.829+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fabd4003970 con 0x7fabdc100210 2024-03-09T00:04:42.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.829+0000 7fabd9ffb640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 0x7fabc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:42.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.830+0000 7fabdbfff640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 0x7fabc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:42.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.830+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fabd40f1210 con 0x7fabdc100210 2024-03-09T00:04:42.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.830+0000 7fabdbfff640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 0x7fabc40a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fabd0006fd0 tx=0x7fabd0008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:42.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:42.834+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fabd4035d60 con 0x7fabdc100210 2024-03-09T00:04:42.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.004+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fabdc0fd930 con 0x7fabc40a2890 2024-03-09T00:04:43.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.014+0000 7fabd9ffb640 1 -- 172.21.15.73:0/2961375941 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340206 (secure 0 0 0) 0x7fabdc0fd930 con 0x7fabc40a2890 2024-03-09T00:04:43.009 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:43.020 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 msgr2=0x7fabc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 0x7fabc40a4d50 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fabd0006fd0 tx=0x7fabd0008040 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 msgr2=0x7fabdc1976a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fabd40324c0 tx=0x7fabd4032f20 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 shutdown_connections 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fabc40a2890 0x7fabc40a4d50 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fabdc101dc0 0x7fabdc19ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fabdc101400 0x7fabdc197be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 --2- 172.21.15.73:0/2961375941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fabdc100210 0x7fabdc1976a0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:43.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 >> 172.21.15.73:0/2961375941 conn(0x7fabdc0fb780 msgr2=0x7fabdc0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:43.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 shutdown_connections 2024-03-09T00:04:43.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:43.018+0000 7fabe3169640 1 -- 172.21.15.73:0/2961375941 wait complete. 2024-03-09T00:04:43.024 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2024-03-09T00:04:41.919495+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":595125,"num_objects":25,"num_object_clones":0,"num_object_copies":76,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":25,"num_whiteouts":0,"num_read":142,"num_read_kb":127,"num_write":231,"num_write_kb":2958,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":181,"ondisk_log_size":181,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":387544,"kb_used_data":13360,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124554280,"statfs":{"total":1151940427776,"available":1151543582720,"internally_reserved":0,"allocated":13680640,"data_stored":7515700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":258,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":3,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003603"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285599+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:04:19.285599+0000","last_peered":"2024-03-09T00:04:19.285599+0000","last_clean":"2024-03-09T00:04:19.285599+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:04:19.285599+0000","last_undegraded":"2024-03-09T00:04:19.285599+0000","last_fullsized":"2024-03-09T00:04:19.285599+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940819+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:04:18.940819+0000","last_peered":"2024-03-09T00:04:18.940819+0000","last_clean":"2024-03-09T00:04:18.940819+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:04:18.940819+0000","last_undegraded":"2024-03-09T00:04:18.940819+0000","last_fullsized":"2024-03-09T00:04:18.940819+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940687+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:04:18.940687+0000","last_peered":"2024-03-09T00:04:18.940687+0000","last_clean":"2024-03-09T00:04:18.940687+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:04:18.940687+0000","last_undegraded":"2024-03-09T00:04:18.940687+0000","last_fullsized":"2024-03-09T00:04:18.940687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284869+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:04:19.284869+0000","last_peered":"2024-03-09T00:04:19.284869+0000","last_clean":"2024-03-09T00:04:19.284869+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:04:19.284869+0000","last_undegraded":"2024-03-09T00:04:19.284869+0000","last_fullsized":"2024-03-09T00:04:19.284869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284695+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:04:19.284695+0000","last_peered":"2024-03-09T00:04:19.284695+0000","last_clean":"2024-03-09T00:04:19.284695+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:04:19.284695+0000","last_undegraded":"2024-03-09T00:04:19.284695+0000","last_fullsized":"2024-03-09T00:04:19.284695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938353+0000","last_change":"2024-03-09T00:04:13.929878+0000","last_active":"2024-03-09T00:04:18.938353+0000","last_peered":"2024-03-09T00:04:18.938353+0000","last_clean":"2024-03-09T00:04:18.938353+0000","last_became_active":"2024-03-09T00:04:13.929508+0000","last_became_peered":"2024-03-09T00:04:13.929508+0000","last_unstale":"2024-03-09T00:04:18.938353+0000","last_undegraded":"2024-03-09T00:04:18.938353+0000","last_fullsized":"2024-03-09T00:04:18.938353+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:51:07.490539+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116022+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:04:19.116022+0000","last_peered":"2024-03-09T00:04:19.116022+0000","last_clean":"2024-03-09T00:04:19.116022+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:04:19.116022+0000","last_undegraded":"2024-03-09T00:04:19.116022+0000","last_fullsized":"2024-03-09T00:04:19.116022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.283878+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:04:19.283878+0000","last_peered":"2024-03-09T00:04:19.283878+0000","last_clean":"2024-03-09T00:04:19.283878+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:04:19.283878+0000","last_undegraded":"2024-03-09T00:04:19.283878+0000","last_fullsized":"2024-03-09T00:04:19.283878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285014+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:04:19.285014+0000","last_peered":"2024-03-09T00:04:19.285014+0000","last_clean":"2024-03-09T00:04:19.285014+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:04:19.285014+0000","last_undegraded":"2024-03-09T00:04:19.285014+0000","last_fullsized":"2024-03-09T00:04:19.285014+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938817+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:04:18.938817+0000","last_peered":"2024-03-09T00:04:18.938817+0000","last_clean":"2024-03-09T00:04:18.938817+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:04:18.938817+0000","last_undegraded":"2024-03-09T00:04:18.938817+0000","last_fullsized":"2024-03-09T00:04:18.938817+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116608+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:04:19.116608+0000","last_peered":"2024-03-09T00:04:19.116608+0000","last_clean":"2024-03-09T00:04:19.116608+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:04:19.116608+0000","last_undegraded":"2024-03-09T00:04:19.116608+0000","last_fullsized":"2024-03-09T00:04:19.116608+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284763+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:04:19.284763+0000","last_peered":"2024-03-09T00:04:19.284763+0000","last_clean":"2024-03-09T00:04:19.284763+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:04:19.284763+0000","last_undegraded":"2024-03-09T00:04:19.284763+0000","last_fullsized":"2024-03-09T00:04:19.284763+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937662+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:04:18.937662+0000","last_peered":"2024-03-09T00:04:18.937662+0000","last_clean":"2024-03-09T00:04:18.937662+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:04:18.937662+0000","last_undegraded":"2024-03-09T00:04:18.937662+0000","last_fullsized":"2024-03-09T00:04:18.937662+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940234+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:04:18.940234+0000","last_peered":"2024-03-09T00:04:18.940234+0000","last_clean":"2024-03-09T00:04:18.940234+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:04:18.940234+0000","last_undegraded":"2024-03-09T00:04:18.940234+0000","last_fullsized":"2024-03-09T00:04:18.940234+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940166+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:04:18.940166+0000","last_peered":"2024-03-09T00:04:18.940166+0000","last_clean":"2024-03-09T00:04:18.940166+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:04:18.940166+0000","last_undegraded":"2024-03-09T00:04:18.940166+0000","last_fullsized":"2024-03-09T00:04:18.940166+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284281+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:04:19.284281+0000","last_peered":"2024-03-09T00:04:19.284281+0000","last_clean":"2024-03-09T00:04:19.284281+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:04:19.284281+0000","last_undegraded":"2024-03-09T00:04:19.284281+0000","last_fullsized":"2024-03-09T00:04:19.284281+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285344+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:04:19.285344+0000","last_peered":"2024-03-09T00:04:19.285344+0000","last_clean":"2024-03-09T00:04:19.285344+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:04:19.285344+0000","last_undegraded":"2024-03-09T00:04:19.285344+0000","last_fullsized":"2024-03-09T00:04:19.285344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284460+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:04:19.284460+0000","last_peered":"2024-03-09T00:04:19.284460+0000","last_clean":"2024-03-09T00:04:19.284460+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:04:19.284460+0000","last_undegraded":"2024-03-09T00:04:19.284460+0000","last_fullsized":"2024-03-09T00:04:19.284460+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286945+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:04:19.286945+0000","last_peered":"2024-03-09T00:04:19.286945+0000","last_clean":"2024-03-09T00:04:19.286945+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:04:19.286945+0000","last_undegraded":"2024-03-09T00:04:19.286945+0000","last_fullsized":"2024-03-09T00:04:19.286945+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286254+0000","last_change":"2024-03-09T00:04:17.955496+0000","last_active":"2024-03-09T00:04:19.286254+0000","last_peered":"2024-03-09T00:04:19.286254+0000","last_clean":"2024-03-09T00:04:19.286254+0000","last_became_active":"2024-03-09T00:04:17.954710+0000","last_became_peered":"2024-03-09T00:04:17.954710+0000","last_unstale":"2024-03-09T00:04:19.286254+0000","last_undegraded":"2024-03-09T00:04:19.286254+0000","last_fullsized":"2024-03-09T00:04:19.286254+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:35:05.318796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,9,0],"acting":[2,3,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":27,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284589+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:04:19.284589+0000","last_peered":"2024-03-09T00:04:19.284589+0000","last_clean":"2024-03-09T00:04:19.284589+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:04:19.284589+0000","last_undegraded":"2024-03-09T00:04:19.284589+0000","last_fullsized":"2024-03-09T00:04:19.284589+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285199+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:04:19.285199+0000","last_peered":"2024-03-09T00:04:19.285199+0000","last_clean":"2024-03-09T00:04:19.285199+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:04:19.285199+0000","last_undegraded":"2024-03-09T00:04:19.285199+0000","last_fullsized":"2024-03-09T00:04:19.285199+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284377+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:04:19.284377+0000","last_peered":"2024-03-09T00:04:19.284377+0000","last_clean":"2024-03-09T00:04:19.284377+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:04:19.284377+0000","last_undegraded":"2024-03-09T00:04:19.284377+0000","last_fullsized":"2024-03-09T00:04:19.284377+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285322+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:04:19.285322+0000","last_peered":"2024-03-09T00:04:19.285322+0000","last_clean":"2024-03-09T00:04:19.285322+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:04:19.285322+0000","last_undegraded":"2024-03-09T00:04:19.285322+0000","last_fullsized":"2024-03-09T00:04:19.285322+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940765+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:04:18.940765+0000","last_peered":"2024-03-09T00:04:18.940765+0000","last_clean":"2024-03-09T00:04:18.940765+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:04:18.940765+0000","last_undegraded":"2024-03-09T00:04:18.940765+0000","last_fullsized":"2024-03-09T00:04:18.940765+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284721+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:04:19.284721+0000","last_peered":"2024-03-09T00:04:19.284721+0000","last_clean":"2024-03-09T00:04:19.284721+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:04:19.284721+0000","last_undegraded":"2024-03-09T00:04:19.284721+0000","last_fullsized":"2024-03-09T00:04:19.284721+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285434+0000","last_change":"2024-03-09T00:04:13.334422+0000","last_active":"2024-03-09T00:04:19.285434+0000","last_peered":"2024-03-09T00:04:19.285434+0000","last_clean":"2024-03-09T00:04:19.285434+0000","last_became_active":"2024-03-09T00:04:13.334101+0000","last_became_peered":"2024-03-09T00:04:13.334101+0000","last_unstale":"2024-03-09T00:04:19.285434+0000","last_undegraded":"2024-03-09T00:04:19.285434+0000","last_fullsized":"2024-03-09T00:04:19.285434+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:04:08.860700+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285247+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:04:19.285247+0000","last_peered":"2024-03-09T00:04:19.285247+0000","last_clean":"2024-03-09T00:04:19.285247+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:04:19.285247+0000","last_undegraded":"2024-03-09T00:04:19.285247+0000","last_fullsized":"2024-03-09T00:04:19.285247+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284658+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:04:19.284658+0000","last_peered":"2024-03-09T00:04:19.284658+0000","last_clean":"2024-03-09T00:04:19.284658+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:04:19.284658+0000","last_undegraded":"2024-03-09T00:04:19.284658+0000","last_fullsized":"2024-03-09T00:04:19.284658+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284812+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:04:19.284812+0000","last_peered":"2024-03-09T00:04:19.284812+0000","last_clean":"2024-03-09T00:04:19.284812+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:04:19.284812+0000","last_undegraded":"2024-03-09T00:04:19.284812+0000","last_fullsized":"2024-03-09T00:04:19.284812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286016+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:04:19.286016+0000","last_peered":"2024-03-09T00:04:19.286016+0000","last_clean":"2024-03-09T00:04:19.286016+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:04:19.286016+0000","last_undegraded":"2024-03-09T00:04:19.286016+0000","last_fullsized":"2024-03-09T00:04:19.286016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286922+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:04:19.286922+0000","last_peered":"2024-03-09T00:04:19.286922+0000","last_clean":"2024-03-09T00:04:19.286922+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:04:19.286922+0000","last_undegraded":"2024-03-09T00:04:19.286922+0000","last_fullsized":"2024-03-09T00:04:19.286922+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285818+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:04:19.285818+0000","last_peered":"2024-03-09T00:04:19.285818+0000","last_clean":"2024-03-09T00:04:19.285818+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:04:19.285818+0000","last_undegraded":"2024-03-09T00:04:19.285818+0000","last_fullsized":"2024-03-09T00:04:19.285818+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284969+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:04:19.284969+0000","last_peered":"2024-03-09T00:04:19.284969+0000","last_clean":"2024-03-09T00:04:19.284969+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:04:19.284969+0000","last_undegraded":"2024-03-09T00:04:19.284969+0000","last_fullsized":"2024-03-09T00:04:19.284969+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286972+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:04:19.286972+0000","last_peered":"2024-03-09T00:04:19.286972+0000","last_clean":"2024-03-09T00:04:19.286972+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:04:19.286972+0000","last_undegraded":"2024-03-09T00:04:19.286972+0000","last_fullsized":"2024-03-09T00:04:19.286972+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938336+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:04:18.938336+0000","last_peered":"2024-03-09T00:04:18.938336+0000","last_clean":"2024-03-09T00:04:18.938336+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:04:18.938336+0000","last_undegraded":"2024-03-09T00:04:18.938336+0000","last_fullsized":"2024-03-09T00:04:18.938336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285928+0000","last_change":"2024-03-09T00:04:13.935057+0000","last_active":"2024-03-09T00:04:19.285928+0000","last_peered":"2024-03-09T00:04:19.285928+0000","last_clean":"2024-03-09T00:04:19.285928+0000","last_became_active":"2024-03-09T00:04:13.934719+0000","last_became_peered":"2024-03-09T00:04:13.934719+0000","last_unstale":"2024-03-09T00:04:19.285928+0000","last_undegraded":"2024-03-09T00:04:19.285928+0000","last_fullsized":"2024-03-09T00:04:19.285928+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:24:41.705525+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285278+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:04:19.285278+0000","last_peered":"2024-03-09T00:04:19.285278+0000","last_clean":"2024-03-09T00:04:19.285278+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:04:19.285278+0000","last_undegraded":"2024-03-09T00:04:19.285278+0000","last_fullsized":"2024-03-09T00:04:19.285278+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937399+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:04:18.937399+0000","last_peered":"2024-03-09T00:04:18.937399+0000","last_clean":"2024-03-09T00:04:18.937399+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:04:18.937399+0000","last_undegraded":"2024-03-09T00:04:18.937399+0000","last_fullsized":"2024-03-09T00:04:18.937399+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284153+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:04:19.284153+0000","last_peered":"2024-03-09T00:04:19.284153+0000","last_clean":"2024-03-09T00:04:19.284153+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:04:19.284153+0000","last_undegraded":"2024-03-09T00:04:19.284153+0000","last_fullsized":"2024-03-09T00:04:19.284153+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940885+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:04:18.940885+0000","last_peered":"2024-03-09T00:04:18.940885+0000","last_clean":"2024-03-09T00:04:18.940885+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:04:18.940885+0000","last_undegraded":"2024-03-09T00:04:18.940885+0000","last_fullsized":"2024-03-09T00:04:18.940885+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284843+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:04:19.284843+0000","last_peered":"2024-03-09T00:04:19.284843+0000","last_clean":"2024-03-09T00:04:19.284843+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:04:19.284843+0000","last_undegraded":"2024-03-09T00:04:19.284843+0000","last_fullsized":"2024-03-09T00:04:19.284843+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284037+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:04:19.284037+0000","last_peered":"2024-03-09T00:04:19.284037+0000","last_clean":"2024-03-09T00:04:19.284037+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:04:19.284037+0000","last_undegraded":"2024-03-09T00:04:19.284037+0000","last_fullsized":"2024-03-09T00:04:19.284037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284237+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:04:19.284237+0000","last_peered":"2024-03-09T00:04:19.284237+0000","last_clean":"2024-03-09T00:04:19.284237+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:04:19.284237+0000","last_undegraded":"2024-03-09T00:04:19.284237+0000","last_fullsized":"2024-03-09T00:04:19.284237+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938748+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:04:18.938748+0000","last_peered":"2024-03-09T00:04:18.938748+0000","last_clean":"2024-03-09T00:04:18.938748+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:04:18.938748+0000","last_undegraded":"2024-03-09T00:04:18.938748+0000","last_fullsized":"2024-03-09T00:04:18.938748+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285004+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:04:19.285004+0000","last_peered":"2024-03-09T00:04:19.285004+0000","last_clean":"2024-03-09T00:04:19.285004+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:04:19.285004+0000","last_undegraded":"2024-03-09T00:04:19.285004+0000","last_fullsized":"2024-03-09T00:04:19.285004+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.283875+0000","last_change":"2024-03-09T00:04:17.960827+0000","last_active":"2024-03-09T00:04:19.283875+0000","last_peered":"2024-03-09T00:04:19.283875+0000","last_clean":"2024-03-09T00:04:19.283875+0000","last_became_active":"2024-03-09T00:04:17.959916+0000","last_became_peered":"2024-03-09T00:04:17.959916+0000","last_unstale":"2024-03-09T00:04:19.283875+0000","last_undegraded":"2024-03-09T00:04:19.283875+0000","last_fullsized":"2024-03-09T00:04:19.283875+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:23:04.325826+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,0,2,4],"acting":[9,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.334152+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.333835+0000","last_became_peered":"2024-03-09T00:04:13.333835+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:19:49.203289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940768+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:04:18.940768+0000","last_peered":"2024-03-09T00:04:18.940768+0000","last_clean":"2024-03-09T00:04:18.940768+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:04:18.940768+0000","last_undegraded":"2024-03-09T00:04:18.940768+0000","last_fullsized":"2024-03-09T00:04:18.940768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285041+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:04:19.285041+0000","last_peered":"2024-03-09T00:04:19.285041+0000","last_clean":"2024-03-09T00:04:19.285041+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:04:19.285041+0000","last_undegraded":"2024-03-09T00:04:19.285041+0000","last_fullsized":"2024-03-09T00:04:19.285041+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285903+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:04:19.285903+0000","last_peered":"2024-03-09T00:04:19.285903+0000","last_clean":"2024-03-09T00:04:19.285903+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:04:19.285903+0000","last_undegraded":"2024-03-09T00:04:19.285903+0000","last_fullsized":"2024-03-09T00:04:19.285903+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285350+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:04:19.285350+0000","last_peered":"2024-03-09T00:04:19.285350+0000","last_clean":"2024-03-09T00:04:19.285350+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:04:19.285350+0000","last_undegraded":"2024-03-09T00:04:19.285350+0000","last_fullsized":"2024-03-09T00:04:19.285350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284040+0000","last_change":"2024-03-09T00:04:17.958243+0000","last_active":"2024-03-09T00:04:19.284040+0000","last_peered":"2024-03-09T00:04:19.284040+0000","last_clean":"2024-03-09T00:04:19.284040+0000","last_became_active":"2024-03-09T00:04:17.957825+0000","last_became_peered":"2024-03-09T00:04:17.957825+0000","last_unstale":"2024-03-09T00:04:19.284040+0000","last_undegraded":"2024-03-09T00:04:19.284040+0000","last_fullsized":"2024-03-09T00:04:19.284040+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:12.927363+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,11,5],"acting":[9,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.14","version":"89'10","reported_seq":37,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:38.992812+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:04:38.992812+0000","last_peered":"2024-03-09T00:04:38.992812+0000","last_clean":"2024-03-09T00:04:38.992812+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:04:38.992812+0000","last_undegraded":"2024-03-09T00:04:38.992812+0000","last_fullsized":"2024-03-09T00:04:38.992812+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":3842,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":4,"num_read_kb":6,"num_write":10,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285266+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:04:19.285266+0000","last_peered":"2024-03-09T00:04:19.285266+0000","last_clean":"2024-03-09T00:04:19.285266+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:04:19.285266+0000","last_undegraded":"2024-03-09T00:04:19.285266+0000","last_fullsized":"2024-03-09T00:04:19.285266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286420+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:04:19.286420+0000","last_peered":"2024-03-09T00:04:19.286420+0000","last_clean":"2024-03-09T00:04:19.286420+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:04:19.286420+0000","last_undegraded":"2024-03-09T00:04:19.286420+0000","last_fullsized":"2024-03-09T00:04:19.286420+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":26,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286747+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:04:19.286747+0000","last_peered":"2024-03-09T00:04:19.286747+0000","last_clean":"2024-03-09T00:04:19.286747+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:04:19.286747+0000","last_undegraded":"2024-03-09T00:04:19.286747+0000","last_fullsized":"2024-03-09T00:04:19.286747+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937495+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:04:18.937495+0000","last_peered":"2024-03-09T00:04:18.937495+0000","last_clean":"2024-03-09T00:04:18.937495+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:04:18.937495+0000","last_undegraded":"2024-03-09T00:04:18.937495+0000","last_fullsized":"2024-03-09T00:04:18.937495+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285804+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:04:19.285804+0000","last_peered":"2024-03-09T00:04:19.285804+0000","last_clean":"2024-03-09T00:04:19.285804+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:04:19.285804+0000","last_undegraded":"2024-03-09T00:04:19.285804+0000","last_fullsized":"2024-03-09T00:04:19.285804+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286331+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:04:19.286331+0000","last_peered":"2024-03-09T00:04:19.286331+0000","last_clean":"2024-03-09T00:04:19.286331+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:04:19.286331+0000","last_undegraded":"2024-03-09T00:04:19.286331+0000","last_fullsized":"2024-03-09T00:04:19.286331+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116312+0000","last_change":"2024-03-09T00:04:13.935230+0000","last_active":"2024-03-09T00:04:19.116312+0000","last_peered":"2024-03-09T00:04:19.116312+0000","last_clean":"2024-03-09T00:04:19.116312+0000","last_became_active":"2024-03-09T00:04:13.934838+0000","last_became_peered":"2024-03-09T00:04:13.934838+0000","last_unstale":"2024-03-09T00:04:19.116312+0000","last_undegraded":"2024-03-09T00:04:19.116312+0000","last_fullsized":"2024-03-09T00:04:19.116312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:37.002289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284230+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:04:19.284230+0000","last_peered":"2024-03-09T00:04:19.284230+0000","last_clean":"2024-03-09T00:04:19.284230+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:04:19.284230+0000","last_undegraded":"2024-03-09T00:04:19.284230+0000","last_fullsized":"2024-03-09T00:04:19.284230+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284338+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:04:19.284338+0000","last_peered":"2024-03-09T00:04:19.284338+0000","last_clean":"2024-03-09T00:04:19.284338+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:04:19.284338+0000","last_undegraded":"2024-03-09T00:04:19.284338+0000","last_fullsized":"2024-03-09T00:04:19.284338+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.115942+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:04:19.115942+0000","last_peered":"2024-03-09T00:04:19.115942+0000","last_clean":"2024-03-09T00:04:19.115942+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:04:19.115942+0000","last_undegraded":"2024-03-09T00:04:19.115942+0000","last_fullsized":"2024-03-09T00:04:19.115942+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937983+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:04:18.937983+0000","last_peered":"2024-03-09T00:04:18.937983+0000","last_clean":"2024-03-09T00:04:18.937983+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:04:18.937983+0000","last_undegraded":"2024-03-09T00:04:18.937983+0000","last_fullsized":"2024-03-09T00:04:18.937983+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":31,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284633+0000","last_change":"2024-03-09T00:04:13.330355+0000","last_active":"2024-03-09T00:04:19.284633+0000","last_peered":"2024-03-09T00:04:19.284633+0000","last_clean":"2024-03-09T00:04:19.284633+0000","last_became_active":"2024-03-09T00:04:13.330012+0000","last_became_peered":"2024-03-09T00:04:13.330012+0000","last_unstale":"2024-03-09T00:04:19.284633+0000","last_undegraded":"2024-03-09T00:04:19.284633+0000","last_fullsized":"2024-03-09T00:04:19.284633+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:46.699807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285690+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:04:19.285690+0000","last_peered":"2024-03-09T00:04:19.285690+0000","last_clean":"2024-03-09T00:04:19.285690+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:04:19.285690+0000","last_undegraded":"2024-03-09T00:04:19.285690+0000","last_fullsized":"2024-03-09T00:04:19.285690+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285653+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:04:19.285653+0000","last_peered":"2024-03-09T00:04:19.285653+0000","last_clean":"2024-03-09T00:04:19.285653+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:04:19.285653+0000","last_undegraded":"2024-03-09T00:04:19.285653+0000","last_fullsized":"2024-03-09T00:04:19.285653+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285009+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:04:19.285009+0000","last_peered":"2024-03-09T00:04:19.285009+0000","last_clean":"2024-03-09T00:04:19.285009+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:04:19.285009+0000","last_undegraded":"2024-03-09T00:04:19.285009+0000","last_fullsized":"2024-03-09T00:04:19.285009+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284962+0000","last_change":"2024-03-09T00:04:13.934570+0000","last_active":"2024-03-09T00:04:19.284962+0000","last_peered":"2024-03-09T00:04:19.284962+0000","last_clean":"2024-03-09T00:04:19.284962+0000","last_became_active":"2024-03-09T00:04:13.934219+0000","last_became_peered":"2024-03-09T00:04:13.934219+0000","last_unstale":"2024-03-09T00:04:19.284962+0000","last_undegraded":"2024-03-09T00:04:19.284962+0000","last_fullsized":"2024-03-09T00:04:19.284962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:37:39.694115+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285000+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:04:19.285000+0000","last_peered":"2024-03-09T00:04:19.285000+0000","last_clean":"2024-03-09T00:04:19.285000+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:04:19.285000+0000","last_undegraded":"2024-03-09T00:04:19.285000+0000","last_fullsized":"2024-03-09T00:04:19.285000+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284300+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:04:19.284300+0000","last_peered":"2024-03-09T00:04:19.284300+0000","last_clean":"2024-03-09T00:04:19.284300+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:04:19.284300+0000","last_undegraded":"2024-03-09T00:04:19.284300+0000","last_fullsized":"2024-03-09T00:04:19.284300+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938530+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:04:18.938530+0000","last_peered":"2024-03-09T00:04:18.938530+0000","last_clean":"2024-03-09T00:04:18.938530+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:04:18.938530+0000","last_undegraded":"2024-03-09T00:04:18.938530+0000","last_fullsized":"2024-03-09T00:04:18.938530+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937720+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:04:18.937720+0000","last_peered":"2024-03-09T00:04:18.937720+0000","last_clean":"2024-03-09T00:04:18.937720+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:04:18.937720+0000","last_undegraded":"2024-03-09T00:04:18.937720+0000","last_fullsized":"2024-03-09T00:04:18.937720+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.938344+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:04:18.938344+0000","last_peered":"2024-03-09T00:04:18.938344+0000","last_clean":"2024-03-09T00:04:18.938344+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:04:18.938344+0000","last_undegraded":"2024-03-09T00:04:18.938344+0000","last_fullsized":"2024-03-09T00:04:18.938344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937783+0000","last_change":"2024-03-09T00:04:18.031380+0000","last_active":"2024-03-09T00:04:18.937783+0000","last_peered":"2024-03-09T00:04:18.937783+0000","last_clean":"2024-03-09T00:04:18.937783+0000","last_became_active":"2024-03-09T00:04:18.030976+0000","last_became_peered":"2024-03-09T00:04:18.030976+0000","last_unstale":"2024-03-09T00:04:18.937783+0000","last_undegraded":"2024-03-09T00:04:18.937783+0000","last_fullsized":"2024-03-09T00:04:18.937783+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:25:18.264867+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,5,10,0],"acting":[6,5,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285050+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:04:19.285050+0000","last_peered":"2024-03-09T00:04:19.285050+0000","last_clean":"2024-03-09T00:04:19.285050+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:04:19.285050+0000","last_undegraded":"2024-03-09T00:04:19.285050+0000","last_fullsized":"2024-03-09T00:04:19.285050+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937260+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:04:18.937260+0000","last_peered":"2024-03-09T00:04:18.937260+0000","last_clean":"2024-03-09T00:04:18.937260+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:04:18.937260+0000","last_undegraded":"2024-03-09T00:04:18.937260+0000","last_fullsized":"2024-03-09T00:04:18.937260+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":15,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:21.234408+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:04:21.234408+0000","last_peered":"2024-03-09T00:04:21.234408+0000","last_clean":"2024-03-09T00:04:21.234408+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:04:21.234408+0000","last_undegraded":"2024-03-09T00:04:21.234408+0000","last_fullsized":"2024-03-09T00:04:21.234408+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284449+0000","last_change":"2024-03-09T00:04:13.939193+0000","last_active":"2024-03-09T00:04:19.284449+0000","last_peered":"2024-03-09T00:04:19.284449+0000","last_clean":"2024-03-09T00:04:19.284449+0000","last_became_active":"2024-03-09T00:04:13.938679+0000","last_became_peered":"2024-03-09T00:04:13.938679+0000","last_unstale":"2024-03-09T00:04:19.284449+0000","last_undegraded":"2024-03-09T00:04:19.284449+0000","last_fullsized":"2024-03-09T00:04:19.284449+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:57:40.949182+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284794+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:04:19.284794+0000","last_peered":"2024-03-09T00:04:19.284794+0000","last_clean":"2024-03-09T00:04:19.284794+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:04:19.284794+0000","last_undegraded":"2024-03-09T00:04:19.284794+0000","last_fullsized":"2024-03-09T00:04:19.284794+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":355,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.115937+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:04:19.115937+0000","last_peered":"2024-03-09T00:04:19.115937+0000","last_clean":"2024-03-09T00:04:19.115937+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:04:19.115937+0000","last_undegraded":"2024-03-09T00:04:19.115937+0000","last_fullsized":"2024-03-09T00:04:19.115937+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285678+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:04:19.285678+0000","last_peered":"2024-03-09T00:04:19.285678+0000","last_clean":"2024-03-09T00:04:19.285678+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:04:19.285678+0000","last_undegraded":"2024-03-09T00:04:19.285678+0000","last_fullsized":"2024-03-09T00:04:19.285678+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937256+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:04:18.937256+0000","last_peered":"2024-03-09T00:04:18.937256+0000","last_clean":"2024-03-09T00:04:18.937256+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:04:18.937256+0000","last_undegraded":"2024-03-09T00:04:18.937256+0000","last_fullsized":"2024-03-09T00:04:18.937256+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116201+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:04:19.116201+0000","last_peered":"2024-03-09T00:04:19.116201+0000","last_clean":"2024-03-09T00:04:19.116201+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:04:19.116201+0000","last_undegraded":"2024-03-09T00:04:19.116201+0000","last_fullsized":"2024-03-09T00:04:19.116201+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284788+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:04:19.284788+0000","last_peered":"2024-03-09T00:04:19.284788+0000","last_clean":"2024-03-09T00:04:19.284788+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:04:19.284788+0000","last_undegraded":"2024-03-09T00:04:19.284788+0000","last_fullsized":"2024-03-09T00:04:19.284788+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285365+0000","last_change":"2024-03-09T00:04:13.331922+0000","last_active":"2024-03-09T00:04:19.285365+0000","last_peered":"2024-03-09T00:04:19.285365+0000","last_clean":"2024-03-09T00:04:19.285365+0000","last_became_active":"2024-03-09T00:04:13.331610+0000","last_became_peered":"2024-03-09T00:04:13.331610+0000","last_unstale":"2024-03-09T00:04:19.285365+0000","last_undegraded":"2024-03-09T00:04:19.285365+0000","last_fullsized":"2024-03-09T00:04:19.285365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:29.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937868+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:04:18.937868+0000","last_peered":"2024-03-09T00:04:18.937868+0000","last_clean":"2024-03-09T00:04:18.937868+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:04:18.937868+0000","last_undegraded":"2024-03-09T00:04:18.937868+0000","last_fullsized":"2024-03-09T00:04:18.937868+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284102+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:04:19.284102+0000","last_peered":"2024-03-09T00:04:19.284102+0000","last_clean":"2024-03-09T00:04:19.284102+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:04:19.284102+0000","last_undegraded":"2024-03-09T00:04:19.284102+0000","last_fullsized":"2024-03-09T00:04:19.284102+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"85'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284558+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:04:19.284558+0000","last_peered":"2024-03-09T00:04:19.284558+0000","last_clean":"2024-03-09T00:04:19.284558+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:04:19.284558+0000","last_undegraded":"2024-03-09T00:04:19.284558+0000","last_fullsized":"2024-03-09T00:04:19.284558+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285217+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:04:19.285217+0000","last_peered":"2024-03-09T00:04:19.285217+0000","last_clean":"2024-03-09T00:04:19.285217+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:04:19.285217+0000","last_undegraded":"2024-03-09T00:04:19.285217+0000","last_fullsized":"2024-03-09T00:04:19.285217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286701+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:04:19.286701+0000","last_peered":"2024-03-09T00:04:19.286701+0000","last_clean":"2024-03-09T00:04:19.286701+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:04:19.286701+0000","last_undegraded":"2024-03-09T00:04:19.286701+0000","last_fullsized":"2024-03-09T00:04:19.286701+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285219+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:04:19.285219+0000","last_peered":"2024-03-09T00:04:19.285219+0000","last_clean":"2024-03-09T00:04:19.285219+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:04:19.285219+0000","last_undegraded":"2024-03-09T00:04:19.285219+0000","last_fullsized":"2024-03-09T00:04:19.285219+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940951+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:04:18.940951+0000","last_peered":"2024-03-09T00:04:18.940951+0000","last_clean":"2024-03-09T00:04:18.940951+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:04:18.940951+0000","last_undegraded":"2024-03-09T00:04:18.940951+0000","last_fullsized":"2024-03-09T00:04:18.940951+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284781+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:04:19.284781+0000","last_peered":"2024-03-09T00:04:19.284781+0000","last_clean":"2024-03-09T00:04:19.284781+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:04:19.284781+0000","last_undegraded":"2024-03-09T00:04:19.284781+0000","last_fullsized":"2024-03-09T00:04:19.284781+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284405+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:04:19.284405+0000","last_peered":"2024-03-09T00:04:19.284405+0000","last_clean":"2024-03-09T00:04:19.284405+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:04:19.284405+0000","last_undegraded":"2024-03-09T00:04:19.284405+0000","last_fullsized":"2024-03-09T00:04:19.284405+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284531+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:04:19.284531+0000","last_peered":"2024-03-09T00:04:19.284531+0000","last_clean":"2024-03-09T00:04:19.284531+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:04:19.284531+0000","last_undegraded":"2024-03-09T00:04:19.284531+0000","last_fullsized":"2024-03-09T00:04:19.284531+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284485+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:04:19.284485+0000","last_peered":"2024-03-09T00:04:19.284485+0000","last_clean":"2024-03-09T00:04:19.284485+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:04:19.284485+0000","last_undegraded":"2024-03-09T00:04:19.284485+0000","last_fullsized":"2024-03-09T00:04:19.284485+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":22,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284303+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:04:19.284303+0000","last_peered":"2024-03-09T00:04:19.284303+0000","last_clean":"2024-03-09T00:04:19.284303+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:04:19.284303+0000","last_undegraded":"2024-03-09T00:04:19.284303+0000","last_fullsized":"2024-03-09T00:04:19.284303+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284537+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:04:19.284537+0000","last_peered":"2024-03-09T00:04:19.284537+0000","last_clean":"2024-03-09T00:04:19.284537+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:04:19.284537+0000","last_undegraded":"2024-03-09T00:04:19.284537+0000","last_fullsized":"2024-03-09T00:04:19.284537+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285237+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:04:19.285237+0000","last_peered":"2024-03-09T00:04:19.285237+0000","last_clean":"2024-03-09T00:04:19.285237+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:04:19.285237+0000","last_undegraded":"2024-03-09T00:04:19.285237+0000","last_fullsized":"2024-03-09T00:04:19.285237+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285989+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:04:19.285989+0000","last_peered":"2024-03-09T00:04:19.285989+0000","last_clean":"2024-03-09T00:04:19.285989+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:04:19.285989+0000","last_undegraded":"2024-03-09T00:04:19.285989+0000","last_fullsized":"2024-03-09T00:04:19.285989+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284711+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:04:19.284711+0000","last_peered":"2024-03-09T00:04:19.284711+0000","last_clean":"2024-03-09T00:04:19.284711+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:04:19.284711+0000","last_undegraded":"2024-03-09T00:04:19.284711+0000","last_fullsized":"2024-03-09T00:04:19.284711+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285355+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:04:19.285355+0000","last_peered":"2024-03-09T00:04:19.285355+0000","last_clean":"2024-03-09T00:04:19.285355+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:04:19.285355+0000","last_undegraded":"2024-03-09T00:04:19.285355+0000","last_fullsized":"2024-03-09T00:04:19.285355+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":25,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284811+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:04:19.284811+0000","last_peered":"2024-03-09T00:04:19.284811+0000","last_clean":"2024-03-09T00:04:19.284811+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:04:19.284811+0000","last_undegraded":"2024-03-09T00:04:19.284811+0000","last_fullsized":"2024-03-09T00:04:19.284811+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116084+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:04:19.116084+0000","last_peered":"2024-03-09T00:04:19.116084+0000","last_clean":"2024-03-09T00:04:19.116084+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:04:19.116084+0000","last_undegraded":"2024-03-09T00:04:19.116084+0000","last_fullsized":"2024-03-09T00:04:19.116084+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284635+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:04:19.284635+0000","last_peered":"2024-03-09T00:04:19.284635+0000","last_clean":"2024-03-09T00:04:19.284635+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:04:19.284635+0000","last_undegraded":"2024-03-09T00:04:19.284635+0000","last_fullsized":"2024-03-09T00:04:19.284635+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":23,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.287034+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:04:19.287034+0000","last_peered":"2024-03-09T00:04:19.287034+0000","last_clean":"2024-03-09T00:04:19.287034+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:04:19.287034+0000","last_undegraded":"2024-03-09T00:04:19.287034+0000","last_fullsized":"2024-03-09T00:04:19.287034+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116620+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:04:19.116620+0000","last_peered":"2024-03-09T00:04:19.116620+0000","last_clean":"2024-03-09T00:04:19.116620+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:04:19.116620+0000","last_undegraded":"2024-03-09T00:04:19.116620+0000","last_fullsized":"2024-03-09T00:04:19.116620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940290+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:04:18.940290+0000","last_peered":"2024-03-09T00:04:18.940290+0000","last_clean":"2024-03-09T00:04:18.940290+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:04:18.940290+0000","last_undegraded":"2024-03-09T00:04:18.940290+0000","last_fullsized":"2024-03-09T00:04:18.940290+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286176+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:04:19.286176+0000","last_peered":"2024-03-09T00:04:19.286176+0000","last_clean":"2024-03-09T00:04:19.286176+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:04:19.286176+0000","last_undegraded":"2024-03-09T00:04:19.286176+0000","last_fullsized":"2024-03-09T00:04:19.286176+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116382+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:04:19.116382+0000","last_peered":"2024-03-09T00:04:19.116382+0000","last_clean":"2024-03-09T00:04:19.116382+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:04:19.116382+0000","last_undegraded":"2024-03-09T00:04:19.116382+0000","last_fullsized":"2024-03-09T00:04:19.116382+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":14,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285655+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:04:19.285655+0000","last_peered":"2024-03-09T00:04:19.285655+0000","last_clean":"2024-03-09T00:04:19.285655+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:04:19.285655+0000","last_undegraded":"2024-03-09T00:04:19.285655+0000","last_fullsized":"2024-03-09T00:04:19.285655+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":24,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937927+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:04:18.937927+0000","last_peered":"2024-03-09T00:04:18.937927+0000","last_clean":"2024-03-09T00:04:18.937927+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:04:18.937927+0000","last_undegraded":"2024-03-09T00:04:18.937927+0000","last_fullsized":"2024-03-09T00:04:18.937927+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.116560+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:04:19.116560+0000","last_peered":"2024-03-09T00:04:19.116560+0000","last_clean":"2024-03-09T00:04:19.116560+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:04:19.116560+0000","last_undegraded":"2024-03-09T00:04:19.116560+0000","last_fullsized":"2024-03-09T00:04:19.116560+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284687+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:04:19.284687+0000","last_peered":"2024-03-09T00:04:19.284687+0000","last_clean":"2024-03-09T00:04:19.284687+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:04:19.284687+0000","last_undegraded":"2024-03-09T00:04:19.284687+0000","last_fullsized":"2024-03-09T00:04:19.284687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286441+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:04:19.286441+0000","last_peered":"2024-03-09T00:04:19.286441+0000","last_clean":"2024-03-09T00:04:19.286441+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:04:19.286441+0000","last_undegraded":"2024-03-09T00:04:19.286441+0000","last_fullsized":"2024-03-09T00:04:19.286441+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285867+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:04:19.285867+0000","last_peered":"2024-03-09T00:04:19.285867+0000","last_clean":"2024-03-09T00:04:19.285867+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:04:19.285867+0000","last_undegraded":"2024-03-09T00:04:19.285867+0000","last_fullsized":"2024-03-09T00:04:19.285867+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.940559+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:04:18.940559+0000","last_peered":"2024-03-09T00:04:18.940559+0000","last_clean":"2024-03-09T00:04:18.940559+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:04:18.940559+0000","last_undegraded":"2024-03-09T00:04:18.940559+0000","last_fullsized":"2024-03-09T00:04:18.940559+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937860+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:04:18.937860+0000","last_peered":"2024-03-09T00:04:18.937860+0000","last_clean":"2024-03-09T00:04:18.937860+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:04:18.937860+0000","last_undegraded":"2024-03-09T00:04:18.937860+0000","last_fullsized":"2024-03-09T00:04:18.937860+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285258+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:04:19.285258+0000","last_peered":"2024-03-09T00:04:19.285258+0000","last_clean":"2024-03-09T00:04:19.285258+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:04:19.285258+0000","last_undegraded":"2024-03-09T00:04:19.285258+0000","last_fullsized":"2024-03-09T00:04:19.285258+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.286463+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:04:19.286463+0000","last_peered":"2024-03-09T00:04:19.286463+0000","last_clean":"2024-03-09T00:04:19.286463+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:04:19.286463+0000","last_undegraded":"2024-03-09T00:04:19.286463+0000","last_fullsized":"2024-03-09T00:04:19.286463+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285311+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:04:19.285311+0000","last_peered":"2024-03-09T00:04:19.285311+0000","last_clean":"2024-03-09T00:04:19.285311+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:04:19.285311+0000","last_undegraded":"2024-03-09T00:04:19.285311+0000","last_fullsized":"2024-03-09T00:04:19.285311+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284953+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:04:19.284953+0000","last_peered":"2024-03-09T00:04:19.284953+0000","last_clean":"2024-03-09T00:04:19.284953+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:04:19.284953+0000","last_undegraded":"2024-03-09T00:04:19.284953+0000","last_fullsized":"2024-03-09T00:04:19.284953+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284416+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:04:19.284416+0000","last_peered":"2024-03-09T00:04:19.284416+0000","last_clean":"2024-03-09T00:04:19.284416+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:04:19.284416+0000","last_undegraded":"2024-03-09T00:04:19.284416+0000","last_fullsized":"2024-03-09T00:04:19.284416+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":21,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.284981+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:04:19.284981+0000","last_peered":"2024-03-09T00:04:19.284981+0000","last_clean":"2024-03-09T00:04:19.284981+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:04:19.284981+0000","last_undegraded":"2024-03-09T00:04:19.284981+0000","last_fullsized":"2024-03-09T00:04:19.284981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:18.937801+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:04:18.937801+0000","last_peered":"2024-03-09T00:04:18.937801+0000","last_clean":"2024-03-09T00:04:18.937801+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:04:18.937801+0000","last_undegraded":"2024-03-09T00:04:18.937801+0000","last_fullsized":"2024-03-09T00:04:18.937801+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":20,"reported_epoch":89,"state":"active+clean","last_fresh":"2024-03-09T00:04:19.285884+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:04:19.285884+0000","last_peered":"2024-03-09T00:04:19.285884+0000","last_clean":"2024-03-09T00:04:19.285884+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:04:19.285884+0000","last_undegraded":"2024-03-09T00:04:19.285884+0000","last_fullsized":"2024-03-09T00:04:19.285884+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":4640,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":6,"num_read_kb":8,"num_write":49,"num_write_kb":19,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":106496,"data_stored":20520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":43,"ondisk_log_size":43,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383699,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36300,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708852,"statfs":{"total":95995035648,"available":95957864448,"internally_reserved":0,"allocated":987136,"data_stored":481311,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645336,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32740,"kb_used_data":1564,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712412,"statfs":{"total":95995035648,"available":95961509888,"internally_reserved":0,"allocated":1601536,"data_stored":1071619,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808861,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36304,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708848,"statfs":{"total":95995035648,"available":95957860352,"internally_reserved":0,"allocated":987136,"data_stored":477797,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070498,"num_pgs":29,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":477819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332135,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32728,"kb_used_data":1544,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712424,"statfs":{"total":95995035648,"available":95961522176,"internally_reserved":0,"allocated":1581056,"data_stored":1068165,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495661,"num_pgs":32,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":983040,"data_stored":477215,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574917,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32148,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713004,"statfs":{"total":95995035648,"available":95962116096,"internally_reserved":0,"allocated":995328,"data_stored":481057,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738442,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27984,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717168,"statfs":{"total":95995035648,"available":95966380032,"internally_reserved":0,"allocated":987136,"data_stored":477305,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411393,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":481091,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182459,"num_pgs":40,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":991232,"data_stored":477319,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920823,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36312,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708840,"statfs":{"total":95995035648,"available":95957852160,"internally_reserved":0,"allocated":987136,"data_stored":477237,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757297,"num_pgs":46,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36896,"kb_used_data":1552,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708256,"statfs":{"total":95995035648,"available":95957254144,"internally_reserved":0,"allocated":1589248,"data_stored":1067765,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":3842,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":10476,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":4036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:04:43.029 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-09T00:04:43.029 INFO:tasks.cephadm:Setting up client nodes... 2024-03-09T00:04:43.029 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-09T00:04:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:43 smithi102 ceph-mon[36106]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:43 smithi102 ceph-mon[36106]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:43 smithi102 ceph-mon[36106]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:43.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1326044312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:43.260 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3851581623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:43.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:43 smithi179 ceph-mon[36637]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:43 smithi179 ceph-mon[36637]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:43 smithi179 ceph-mon[36637]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:43.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1326044312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:43.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3851581623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:43 smithi073 ceph-mon[32309]: from='client.14916 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:43 smithi073 ceph-mon[32309]: from='client.14922 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:04:43.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:43 smithi073 ceph-mon[32309]: pgmap v308: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 682 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1326044312' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:04:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3851581623' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:44.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:44 smithi102 ceph-mon[36106]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:44.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:44 smithi102 ceph-mon[36106]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:44.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:44 smithi179 ceph-mon[36637]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:44.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:44 smithi179 ceph-mon[36637]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:44.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:44 smithi073 ceph-mon[32309]: from='client.14940 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:44.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:44 smithi073 ceph-mon[32309]: from='client.14946 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:04:45.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 -- 172.21.15.73:0/744839738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 msgr2=0x7f23f0101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 --2- 172.21.15.73:0/744839738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0101140 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f23ec0099c0 tx=0x7f23ec009cd0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 -- 172.21.15.73:0/744839738 shutdown_connections 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 --2- 172.21.15.73:0/744839738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0101140 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 --2- 172.21.15.73:0/744839738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f01028c0 0x7f23f0109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 --2- 172.21.15.73:0/744839738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f0101f20 0x7f23f0102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 -- 172.21.15.73:0/744839738 >> 172.21.15.73:0/744839738 conn(0x7f23f00fc2e0 msgr2=0x7f23f00fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 -- 172.21.15.73:0/744839738 shutdown_connections 2024-03-09T00:04:45.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.082+0000 7f23f807a700 1 -- 172.21.15.73:0/744839738 wait complete. 2024-03-09T00:04:45.078 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:45 smithi073 ceph-mon[32309]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 Processor -- start 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 -- start start 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f0101f20 0x7f23f0198500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f01028c0 0x7f23f019f580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:45.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f010c960 con 0x7f23f0100d20 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f010c7e0 con 0x7f23f0101f20 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f807a700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23f010cae0 con 0x7f23f01028c0 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f5e16700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f5e16700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35650/0 (socket says 172.21.15.73:35650) 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f5e16700 1 -- 172.21.15.73:0/3611041435 learned_addr learned my addr 172.21.15.73:0/3611041435 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.083+0000 7f23f5615700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f0101f20 0x7f23f0198500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f6617700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f01028c0 0x7f23f019f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 -- 172.21.15.73:0/3611041435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f01028c0 msgr2=0x7f23f019f580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f01028c0 0x7f23f019f580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 -- 172.21.15.73:0/3611041435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f0101f20 msgr2=0x7f23f0198500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f0101f20 0x7f23f0198500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 -- 172.21.15.73:0/3611041435 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23f019fc20 con 0x7f23f0100d20 2024-03-09T00:04:45.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f5e16700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec0057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:45.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec027070 con 0x7f23f0100d20 2024-03-09T00:04:45.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f23ec004150 con 0x7f23f0100d20 2024-03-09T00:04:45.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23f019feb0 con 0x7f23f0100d20 2024-03-09T00:04:45.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.084+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23f01a02d0 con 0x7f23f0100d20 2024-03-09T00:04:45.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.085+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f23ec00f460 con 0x7f23f0100d20 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.086+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23f00502f0 con 0x7f23f0100d20 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.087+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f23ec02a070 con 0x7f23f0100d20 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.087+0000 7f23e6ffd700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 0x7f23dc0a4e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.088+0000 7f23f5615700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 0x7f23dc0a4e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.088+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f23ec0cfe90 con 0x7f23f0100d20 2024-03-09T00:04:45.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.088+0000 7f23f5615700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 0x7f23dc0a4e90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009a10 tx=0x7f23e0009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:45.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.092+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f23ec013070 con 0x7f23f0100d20 2024-03-09T00:04:45.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:45 smithi102 ceph-mon[36106]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:45 smithi179 ceph-mon[36637]: pgmap v309: 129 pgs: 129 active+clean; 581 KiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 0 op/s 2024-03-09T00:04:45.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.382+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f23f01a08c0 con 0x7f23f0100d20 2024-03-09T00:04:45.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.385+0000 7f23e6ffd700 1 -- 172.21.15.73:0/3611041435 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f23f01a08c0 con 0x7f23f0100d20 2024-03-09T00:04:45.380 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2024-03-09T00:04:45.380 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQCdp+tlMiLoFhAAGBJx8F5JjNHf6sRJASd6ZQ== 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 msgr2=0x7f23dc0a4e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 0x7f23dc0a4e90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f23e0009a10 tx=0x7f23e0009190 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 msgr2=0x7f23f0197fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f23ec006010 tx=0x7f23ec0057c0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 shutdown_connections 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f23f0100d20 0x7f23f0197fc0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f23dc0a29d0 0x7f23dc0a4e90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f23f01028c0 0x7f23f019f580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 --2- 172.21.15.73:0/3611041435 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f23f0101f20 0x7f23f0198500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:45.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 >> 172.21.15.73:0/3611041435 conn(0x7f23f00fc2e0 msgr2=0x7f23f00fdcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:45.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 shutdown_connections 2024-03-09T00:04:45.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:45.387+0000 7f23f807a700 1 -- 172.21.15.73:0/3611041435 wait complete. 2024-03-09T00:04:46.039 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:04:46.039 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-09T00:04:46.040 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-09T00:04:46.083 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-09T00:04:46.083 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-09T00:04:46.084 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph mgr dump --format=json 2024-03-09T00:04:46.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:46 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-09T00:04:46.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:46 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-09T00:04:46.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:46 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-09T00:04:46.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:46 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-09T00:04:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:46 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-09T00:04:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:46 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3611041435' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-09T00:04:46.370 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:47.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:47 smithi102 ceph-mon[36106]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-03-09T00:04:47.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:47 smithi179 ceph-mon[36637]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-03-09T00:04:47.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:47 smithi073 ceph-mon[32309]: pgmap v310: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.3 KiB/s wr, 3 op/s 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.281+0000 7ff4c35d7700 1 -- 172.21.15.73:0/3945312559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc100f70 msgr2=0x7ff4bc101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.281+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/3945312559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc100f70 0x7ff4bc101350 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac009990 tx=0x7ff4ac009ca0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 -- 172.21.15.73:0/3945312559 shutdown_connections 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/3945312559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc100f70 0x7ff4bc101350 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/3945312559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc106000 0x7ff4bc10dbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/3945312559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc101920 0x7ff4bc105920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 -- 172.21.15.73:0/3945312559 >> 172.21.15.73:0/3945312559 conn(0x7ff4bc076d00 msgr2=0x7ff4bc077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 -- 172.21.15.73:0/3945312559 shutdown_connections 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 -- 172.21.15.73:0/3945312559 wait complete. 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 Processor -- start 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.282+0000 7ff4c35d7700 1 -- start start 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc100f70 0x7ff4bc199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc106000 0x7ff4bc1a4500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc111020 con 0x7ff4bc101920 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc110ea0 con 0x7ff4bc106000 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4bc1111a0 con 0x7ff4bc100f70 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:48.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:35668/0 (socket says 172.21.15.73:35668) 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 -- 172.21.15.73:0/843443617 learned_addr learned my addr 172.21.15.73:0/843443617 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c1b74700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc106000 0x7ff4bc1a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c1373700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc100f70 0x7ff4bc199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 -- 172.21.15.73:0/843443617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc100f70 msgr2=0x7ff4bc199480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc100f70 0x7ff4bc199480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 -- 172.21.15.73:0/843443617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc106000 msgr2=0x7ff4bc1a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc106000 0x7ff4bc1a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 -- 172.21.15.73:0/843443617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc1a4a40 con 0x7ff4bc101920 2024-03-09T00:04:48.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c0b72700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800d980 tx=0x7ff4b800dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:48.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b8017070 con 0x7ff4bc101920 2024-03-09T00:04:48.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4bc1a4c70 con 0x7ff4bc101920 2024-03-09T00:04:48.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff4b800fd70 con 0x7ff4bc101920 2024-03-09T00:04:48.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.283+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff4b800e3b0 con 0x7ff4bc101920 2024-03-09T00:04:48.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.284+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4bc1a5180 con 0x7ff4bc101920 2024-03-09T00:04:48.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.285+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4bc102400 con 0x7ff4bc101920 2024-03-09T00:04:48.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.286+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff4b800f890 con 0x7ff4bc101920 2024-03-09T00:04:48.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.287+0000 7ff4b27fc700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 0x7ff4a80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:48.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.287+0000 7ff4c1373700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 0x7ff4a80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:48.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.287+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7ff4b80ce840 con 0x7ff4bc101920 2024-03-09T00:04:48.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.287+0000 7ff4c1373700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 0x7ff4a80a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00b5c0 tx=0x7ff4ac005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:48.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.290+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff4b809b540 con 0x7ff4bc101920 2024-03-09T00:04:48.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.548+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7ff4bc02d9f0 con 0x7ff4bc101920 2024-03-09T00:04:48.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.553+0000 7ff4b27fc700 1 -- 172.21.15.73:0/843443617 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262660 (secure 0 0 0) 0x7ff4b809ae40 con 0x7ff4bc101920 2024-03-09T00:04:48.551 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 msgr2=0x7ff4a80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 0x7ff4a80a4de0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ff4ac00b5c0 tx=0x7ff4ac005960 comp rx=0 tx=0).stop 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 msgr2=0x7ff4bc1999c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7ff4b800d980 tx=0x7ff4b800dc90 comp rx=0 tx=0).stop 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 shutdown_connections 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff4a80a2920 0x7ff4a80a4de0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff4bc106000 0x7ff4bc1a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff4bc100f70 0x7ff4bc199480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 --2- 172.21.15.73:0/843443617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff4bc101920 0x7ff4bc1999c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:48.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 >> 172.21.15.73:0/843443617 conn(0x7ff4bc076d00 msgr2=0x7ff4bc1004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:48.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 shutdown_connections 2024-03-09T00:04:48.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:48.556+0000 7ff4c35d7700 1 -- 172.21.15.73:0/843443617 wait complete. 2024-03-09T00:04:49.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:49 smithi179 ceph-mon[36637]: pgmap v311: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-09T00:04:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:49 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/843443617' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-09T00:04:49.352 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:49 smithi073 ceph-mon[32309]: pgmap v311: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-09T00:04:49.352 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:49 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/843443617' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-09T00:04:49.385 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":2701889057},{"type":"v1","addr":"172.21.15.73:6801","nonce":2701889057}]},"active_addr":"172.21.15.73:6801/2701889057","active_change":"2024-03-08T23:57:03.831172+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24115,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24122,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.73:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":102170643}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":2748563172}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3270366724}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":2159650606}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":4030685318}]}]} 2024-03-09T00:04:49.389 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-09T00:04:49.389 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-09T00:04:49.390 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-09T00:04:49.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:49 smithi102 ceph-mon[36106]: pgmap v311: 129 pgs: 129 active+clean; 622 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 4.9 KiB/s rd, 4.2 KiB/s wr, 3 op/s 2024-03-09T00:04:49.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:49 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/843443617' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-09T00:04:49.636 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:51.072 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:51 smithi073 ceph-mon[32309]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-09T00:04:51.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:51 smithi179 ceph-mon[36637]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-09T00:04:51.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/1037661324 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84102250 msgr2=0x7fdb8410dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:51.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/1037661324 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84102250 0x7fdb8410dc00 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdb8000b0d0 tx=0x7fdb8000b3e0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/1037661324 shutdown_connections 2024-03-09T00:04:51.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/1037661324 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84102250 0x7fdb8410dc00 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/1037661324 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84105d10 0x7fdb841060f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/1037661324 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84101890 0x7fdb84101d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.420+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/1037661324 >> 172.21.15.73:0/1037661324 conn(0x7fdb84076060 msgr2=0x7fdb84076470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/1037661324 shutdown_connections 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/1037661324 wait complete. 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 Processor -- start 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- start start 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84101890 0x7fdb84197160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84102250 0x7fdb841976a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb84111090 con 0x7fdb84105d10 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb84110f10 con 0x7fdb84102250 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8c6b5700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb84111210 con 0x7fdb84101890 2024-03-09T00:04:51.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8ac52700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8ac52700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57676/0 (socket says 172.21.15.73:57676) 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb8ac52700 1 -- 172.21.15.73:0/3305265364 learned_addr learned my addr 172.21.15.73:0/3305265364 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.421+0000 7fdb89c50700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84102250 0x7fdb841976a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8a451700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84101890 0x7fdb84197160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 -- 172.21.15.73:0/3305265364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84101890 msgr2=0x7fdb84197160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84101890 0x7fdb84197160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 -- 172.21.15.73:0/3305265364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84102250 msgr2=0x7fdb841976a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84102250 0x7fdb841976a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 -- 172.21.15.73:0/3305265364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb841a69d0 con 0x7fdb84105d10 2024-03-09T00:04:51.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8ac52700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdb80006070 tx=0x7fdb80004d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb80021070 con 0x7fdb84105d10 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdb80003c00 con 0x7fdb84105d10 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb841a6c60 con 0x7fdb84105d10 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdb80004830 con 0x7fdb84105d10 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.422+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb841a70c0 con 0x7fdb84105d10 2024-03-09T00:04:51.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.423+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb84068aa0 con 0x7fdb84105d10 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.424+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdb80003db0 con 0x7fdb84105d10 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.425+0000 7fdb7b7fe700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 0x7fdb700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.425+0000 7fdb8a451700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 0x7fdb700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.425+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7fdb800cfd70 con 0x7fdb84105d10 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.425+0000 7fdb8a451700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 0x7fdb700a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005950 tx=0x7fdb7c00a300 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:51.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.429+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdb8009cb00 con 0x7fdb84105d10 2024-03-09T00:04:51.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:51 smithi102 ceph-mon[36106]: pgmap v312: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 5.9 KiB/s wr, 5 op/s 2024-03-09T00:04:51.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.624+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdb8402d830 con 0x7fdb84105d10 2024-03-09T00:04:51.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.625+0000 7fdb7b7fe700 1 -- 172.21.15.73:0/3305265364 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7fdb8009c400 con 0x7fdb84105d10 2024-03-09T00:04:51.621 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:51.621 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:51.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 msgr2=0x7fdb700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:51.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 0x7fdb700a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fdb7c005950 tx=0x7fdb7c00a300 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 msgr2=0x7fdb84197be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdb80006070 tx=0x7fdb80004d50 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 shutdown_connections 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdb84105d10 0x7fdb84197be0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdb700a2920 0x7fdb700a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdb84101890 0x7fdb84197160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 --2- 172.21.15.73:0/3305265364 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdb84102250 0x7fdb841976a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.627+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 >> 172.21.15.73:0/3305265364 conn(0x7fdb84076060 msgr2=0x7fdb840ff3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.628+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 shutdown_connections 2024-03-09T00:04:51.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:51.628+0000 7fdb8c6b5700 1 -- 172.21.15.73:0/3305265364 wait complete. 2024-03-09T00:04:52.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:52 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3305265364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:52 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3305265364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:52.444 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-09T00:04:52.444 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd dump --format=json 2024-03-09T00:04:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:52 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3305265364' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:52.688 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:53.063 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:53 smithi073 ceph-mon[32309]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-09T00:04:53.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:53 smithi179 ceph-mon[36637]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-09T00:04:53.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:53 smithi102 ceph-mon[36106]: pgmap v313: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 5.9 KiB/s wr, 4 op/s 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 -- 172.21.15.73:0/2761540675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 msgr2=0x7f609c10db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 --2- 172.21.15.73:0/2761540675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c10db60 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f609400b0a0 tx=0x7f609400b3b0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 -- 172.21.15.73:0/2761540675 shutdown_connections 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 --2- 172.21.15.73:0/2761540675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c10db60 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 --2- 172.21.15.73:0/2761540675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 --2- 172.21.15.73:0/2761540675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 0x7f609c101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 -- 172.21.15.73:0/2761540675 >> 172.21.15.73:0/2761540675 conn(0x7f609c077c10 msgr2=0x7f609c078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 -- 172.21.15.73:0/2761540675 shutdown_connections 2024-03-09T00:04:54.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.565+0000 7f60a2958700 1 -- 172.21.15.73:0/2761540675 wait complete. 2024-03-09T00:04:54.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 Processor -- start 2024-03-09T00:04:54.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 -- start start 2024-03-09T00:04:54.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 0x7f609c102990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:54.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:54.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c103410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:54.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c110fa0 con 0x7f609c101680 2024-03-09T00:04:54.563 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c110e20 con 0x7f609c106d20 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.566+0000 7f60a2958700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f609c111120 con 0x7f609c100ce0 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57704/0 (socket says 172.21.15.73:57704) 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 -- 172.21.15.73:0/4148962726 learned_addr learned my addr 172.21.15.73:0/4148962726 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609bfff700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 0x7f609c102990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f60a0ef5700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c103410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 -- 172.21.15.73:0/4148962726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 msgr2=0x7f609c102990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 0x7f609c102990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 -- 172.21.15.73:0/4148962726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 msgr2=0x7f609c103410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c103410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 -- 172.21.15.73:0/4148962726 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c1a69c0 con 0x7f609c101680 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f60a0ef5700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:04:54.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f609b7fe700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f609000d980 tx=0x7f609000dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090017070 con 0x7f609c101680 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f609c1a6cb0 con 0x7f609c101680 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.567+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6090005020 con 0x7f609c101680 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.568+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6090009bc0 con 0x7f609c101680 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.568+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f609c1a71a0 con 0x7f609c101680 2024-03-09T00:04:54.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.569+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f609c068aa0 con 0x7f609c101680 2024-03-09T00:04:54.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.570+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f60900051d0 con 0x7f609c101680 2024-03-09T00:04:54.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.570+0000 7f60997fa700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 0x7f60840a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:04:54.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.570+0000 7f609bfff700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 0x7f60840a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:04:54.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.570+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f60900ceca0 con 0x7f609c101680 2024-03-09T00:04:54.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.571+0000 7f609bfff700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 0x7f60840a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f608c009be0 tx=0x7f608c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:04:54.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.575+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6090020020 con 0x7f609c101680 2024-03-09T00:04:54.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.774+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f609c1a74d0 con 0x7f609c101680 2024-03-09T00:04:54.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.775+0000 7f60997fa700 1 -- 172.21.15.73:0/4148962726 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+24115 (secure 0 0 0) 0x7f609009b9a0 con 0x7f609c101680 2024-03-09T00:04:54.771 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:04:54.771 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":89,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:04:18.298551+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":83,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 msgr2=0x7f60840a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 0x7f60840a4e60 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f608c009be0 tx=0x7f608c009260 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 msgr2=0x7f609c102ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f609000d980 tx=0x7f609000dc90 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 shutdown_connections 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.777+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60840a29a0 0x7f60840a4e60 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f609c106d20 0x7f609c103410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f609c100ce0 0x7f609c102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 --2- 172.21.15.73:0/4148962726 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f609c101680 0x7f609c102ed0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 >> 172.21.15.73:0/4148962726 conn(0x7f609c077c10 msgr2=0x7f609c0fff60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 shutdown_connections 2024-03-09T00:04:54.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:04:54.778+0000 7f60a2958700 1 -- 172.21.15.73:0/4148962726 wait complete. 2024-03-09T00:04:55.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:55 smithi179 ceph-mon[36637]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.8 KiB/s wr, 4 op/s 2024-03-09T00:04:55.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:55 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/4148962726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:55 smithi073 ceph-mon[32309]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.8 KiB/s wr, 4 op/s 2024-03-09T00:04:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:55 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/4148962726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:55 smithi102 ceph-mon[36106]: pgmap v314: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 5.8 KiB/s wr, 4 op/s 2024-03-09T00:04:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:55 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/4148962726' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:04:55.623 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-09T00:04:55.623 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-09T00:04:55.623 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-09T00:04:55.624 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-09T00:04:55.624 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-09T00:04:55.625 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-09T00:04:55.625 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-09T00:04:55.626 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-09T00:04:55.626 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-09T00:04:55.627 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-09T00:04:55.627 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-09T00:04:55.628 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-09T00:04:56.323 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:56.324 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:56.326 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:56.336 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:56.337 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:04:57.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:57 smithi179 ceph-mon[36637]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-09T00:04:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:57 smithi073 ceph-mon[32309]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-09T00:04:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:57 smithi102 ceph-mon[36106]: pgmap v315: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 5.8 KiB/s wr, 5 op/s 2024-03-09T00:04:59.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:04:59 smithi179 ceph-mon[36637]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-09T00:04:59.339 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:04:59 smithi073 ceph-mon[32309]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-09T00:04:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:04:59 smithi102 ceph-mon[36106]: pgmap v316: 129 pgs: 129 active+clean; 638 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.7 KiB/s wr, 2 op/s 2024-03-09T00:05:01.210 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:01 smithi073 ceph-mon[32309]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-09T00:05:01.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 -- 172.21.15.73:0/2572454629 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700101e60 msgr2=0x7f17001022e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 --2- 172.21.15.73:0/2572454629 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700101e60 0x7f17001022e0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f16f4009990 tx=0x7f16f4009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 -- 172.21.15.73:0/2572454629 shutdown_connections 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 --2- 172.21.15.73:0/2572454629 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1700102820 0x7f170010ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 --2- 172.21.15.73:0/2572454629 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700107eb0 0x7f1700108290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 --2- 172.21.15.73:0/2572454629 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700101e60 0x7f17001022e0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 -- 172.21.15.73:0/2572454629 >> 172.21.15.73:0/2572454629 conn(0x7f1700078f30 msgr2=0x7f1700079340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 -- 172.21.15.73:0/2572454629 shutdown_connections 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 -- 172.21.15.73:0/2572454629 wait complete. 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.215+0000 7f1706313700 1 Processor -- start 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 -- start start 2024-03-09T00:05:01.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1700101e60 0x7f1700198580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700102820 0x7f1700198ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 0x7f170019ce50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001120d0 con 0x7f1700107eb0 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700111f50 con 0x7f1700101e60 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f1706313700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700112250 con 0x7f1700102820 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 0x7f170019ce50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 0x7f170019ce50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53500/0 (socket says 172.21.15.73:53500) 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 -- 172.21.15.73:0/1048857676 learned_addr learned my addr 172.21.15.73:0/1048857676 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f16fffff700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1700101e60 0x7f1700198580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f16ff7fe700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700102820 0x7f1700198ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 -- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700102820 msgr2=0x7f1700198ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700102820 0x7f1700198ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 -- 172.21.15.73:0/1048857676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1700101e60 msgr2=0x7f1700198580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1700101e60 0x7f1700198580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f17048b0700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170019d590 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.216+0000 7f16ff7fe700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1700102820 0x7f1700198ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f17048b0700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 0x7f170019ce50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16f800a110 tx=0x7f16f800a420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f801d070 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f170019d880 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f16f800ad60 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1700074010 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.217+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f16f8017790 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.218+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f17000624f0 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.219+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f16f8020070 con 0x7f1700107eb0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.219+0000 7f16fd7fa700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f16e80a2920 0x7f16e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fffff700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f16e80a2920 0x7f16e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:01.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f16f80d2a20 con 0x7f1700107eb0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fd7fa700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] conn(0x7f16e80ad5e0 0x7f16e80afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f16e80b00f0 con 0x7f16e80ad5e0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f16f80d2e70 con 0x7f1700107eb0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16fffff700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f16e80a2920 0x7f16e80a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f16f0005950 tx=0x7f16f000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.220+0000 7f16ff7fe700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] conn(0x7f16e80ad5e0 0x7f16e80afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.221+0000 7f16ff7fe700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] conn(0x7f16e80ad5e0 0x7f16e80afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:01.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.222+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== osd.5 v2:172.21.15.179:6808/833646001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f16e80b00f0 con 0x7f16e80ad5e0 2024-03-09T00:05:01.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.238+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 --> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f17000628f0 con 0x7f16e80ad5e0 2024-03-09T00:05:01.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.238+0000 7f16fd7fa700 1 -- 172.21.15.73:0/1048857676 <== osd.5 v2:172.21.15.179:6808/833646001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f17000628f0 con 0x7f16e80ad5e0 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.238+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] conn(0x7f16e80ad5e0 msgr2=0x7f16e80afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.238+0000 7f1706313700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] conn(0x7f16e80ad5e0 0x7f16e80afa20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f16e80a2920 msgr2=0x7f16e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f16e80a2920 0x7f16e80a4de0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f16f0005950 tx=0x7f16f000b410 comp rx=0 tx=0).stop 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 msgr2=0x7f170019ce50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 --2- 172.21.15.73:0/1048857676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1700107eb0 0x7f170019ce50 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f16f800a110 tx=0x7f16f800a420 comp rx=0 tx=0).stop 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f17048b0700 1 -- 172.21.15.73:0/1048857676 reap_dead start 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 shutdown_connections 2024-03-09T00:05:01.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 >> 172.21.15.73:0/1048857676 conn(0x7f1700078f30 msgr2=0x7f1700100bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:01.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 shutdown_connections 2024-03-09T00:05:01.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:01.239+0000 7f1706313700 1 -- 172.21.15.73:0/1048857676 wait complete. 2024-03-09T00:05:01.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:01 smithi179 ceph-mon[36637]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-09T00:05:01.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:01 smithi102 ceph-mon[36106]: pgmap v317: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.9 KiB/s wr, 3 op/s 2024-03-09T00:05:02.183 INFO:teuthology.orchestra.run.smithi073.stdout:163208757302 2024-03-09T00:05:02.183 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-09T00:05:02.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/149224036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 msgr2=0x7f8d38106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/149224036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38106a20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c0099c0 tx=0x7f8d2c009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/149224036 shutdown_connections 2024-03-09T00:05:02.373 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/149224036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d38107100 0x7f8d3810ecd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/149224036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d38102020 0x7f8d38102400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/149224036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38106a20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/149224036 >> 172.21.15.73:0/149224036 conn(0x7f8d380fd890 msgr2=0x7f8d380ffcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:02.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/149224036 shutdown_connections 2024-03-09T00:05:02.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.378+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/149224036 wait complete. 2024-03-09T00:05:02.375 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 Processor -- start 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 -- start start 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d38102020 0x7f8d380740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d38107100 0x7f8d38074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d38112140 con 0x7f8d381029d0 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d38111fc0 con 0x7f8d38102020 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3f4ab700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d381122c0 con 0x7f8d38107100 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3ca46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3ca46700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53518/0 (socket says 172.21.15.73:53518) 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3ca46700 1 -- 172.21.15.73:0/2245017888 learned_addr learned my addr 172.21.15.73:0/2245017888 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3d247700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d38102020 0x7f8d380740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.379+0000 7f8d3da48700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d38107100 0x7f8d38074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 -- 172.21.15.73:0/2245017888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d38107100 msgr2=0x7f8d38074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8d38107100 0x7f8d38074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 -- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d38102020 msgr2=0x7f8d380740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8d38102020 0x7f8d380740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d381a6af0 con 0x7f8d381029d0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3ca46700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38074620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00c010 tx=0x7f8d2c0047b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c004830 con 0x7f8d381029d0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d381a6d80 con 0x7f8d381029d0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d2c005020 con 0x7f8d381029d0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d2c003cb0 con 0x7f8d381029d0 2024-03-09T00:05:02.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.380+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d381a7180 con 0x7f8d381029d0 2024-03-09T00:05:02.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.381+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8d380624b0 con 0x7f8d381029d0 2024-03-09T00:05:02.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.382+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8d2c0049d0 con 0x7f8d381029d0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d367fc700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8d200a2920 0x7f8d200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8802+0+0 (secure 0 0 0) 0x7f8d2c0cfcb0 con 0x7f8d381029d0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d3d247700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8d200a2920 0x7f8d200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d367fc700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] conn(0x7f8d200ad5e0 0x7f8d200afa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8d200b00f0 con 0x7f8d200ad5e0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=89) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d2c0d2370 con 0x7f8d381029d0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d3d247700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8d200a2920 0x7f8d200a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000a400 tx=0x7f8d3000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.383+0000 7f8d3da48700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] conn(0x7f8d200ad5e0 0x7f8d200afa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.384+0000 7f8d3da48700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] conn(0x7f8d200ad5e0 0x7f8d200afa20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:02.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.385+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== osd.4 v2:172.21.15.102:6808/1213471038 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8d200b00f0 con 0x7f8d200ad5e0 2024-03-09T00:05:02.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.407+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 --> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8d38062960 con 0x7f8d200ad5e0 2024-03-09T00:05:02.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.408+0000 7f8d367fc700 1 -- 172.21.15.73:0/2245017888 <== osd.4 v2:172.21.15.102:6808/1213471038 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8d38062960 con 0x7f8d200ad5e0 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.408+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] conn(0x7f8d200ad5e0 msgr2=0x7f8d200afa20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.408+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] conn(0x7f8d200ad5e0 0x7f8d200afa20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8d200a2920 msgr2=0x7f8d200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f8d200a2920 0x7f8d200a4de0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8d3000a400 tx=0x7f8d3000d040 comp rx=0 tx=0).stop 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 msgr2=0x7f8d38074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 --2- 172.21.15.73:0/2245017888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8d381029d0 0x7f8d38074620 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f8d2c00c010 tx=0x7f8d2c0047b0 comp rx=0 tx=0).stop 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3da48700 1 -- 172.21.15.73:0/2245017888 reap_dead start 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 shutdown_connections 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 >> 172.21.15.73:0/2245017888 conn(0x7f8d380fd890 msgr2=0x7f8d380ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 shutdown_connections 2024-03-09T00:05:02.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:02.409+0000 7f8d3f4ab700 1 -- 172.21.15.73:0/2245017888 wait complete. 2024-03-09T00:05:03.123 INFO:teuthology.orchestra.run.smithi073.stdout:141733920828 2024-03-09T00:05:03.124 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-09T00:05:03.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:03 smithi179 ceph-mon[36637]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:03.356 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:03 smithi073 ceph-mon[32309]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:03.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:03 smithi102 ceph-mon[36106]: pgmap v318: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-09T00:05:04.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:04 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-09T00:05:04.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-09T00:05:04.366 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:04 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 -- 172.21.15.73:0/614421878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc100ce0 msgr2=0x7f14dc101140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 --2- 172.21.15.73:0/614421878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc100ce0 0x7f14dc101140 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7f14c4009990 tx=0x7f14c4009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 -- 172.21.15.73:0/614421878 shutdown_connections 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 --2- 172.21.15.73:0/614421878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc101680 0x7f14dc10db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 --2- 172.21.15.73:0/614421878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14dc106d20 0x7f14dc107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 --2- 172.21.15.73:0/614421878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc100ce0 0x7f14dc101140 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 -- 172.21.15.73:0/614421878 >> 172.21.15.73:0/614421878 conn(0x7f14dc077c10 msgr2=0x7f14dc078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 -- 172.21.15.73:0/614421878 shutdown_connections 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.471+0000 7f14e1055700 1 -- 172.21.15.73:0/614421878 wait complete. 2024-03-09T00:05:04.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.472+0000 7f14e1055700 1 Processor -- start 2024-03-09T00:05:04.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.472+0000 7f14e1055700 1 -- start start 2024-03-09T00:05:04.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.472+0000 7f14e1055700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14dc100ce0 0x7f14dc198530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:04.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.472+0000 7f14e1055700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc101680 0x7f14dc198a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:04.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.472+0000 7f14e1055700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 0x7f14dc19ce00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14e1055700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc110f80 con 0x7f14dc106d20 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14e1055700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc110e00 con 0x7f14dc101680 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14e1055700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14dc111100 con 0x7f14dc100ce0 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 0x7f14dc19ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 0x7f14dc19ce00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53540/0 (socket says 172.21.15.73:53540) 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 -- 172.21.15.73:0/3390139569 learned_addr learned my addr 172.21.15.73:0/3390139569 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14da59c700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc101680 0x7f14dc198a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14dad9d700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14dc100ce0 0x7f14dc198530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 -- 172.21.15.73:0/3390139569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14dc100ce0 msgr2=0x7f14dc198530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f14dc100ce0 0x7f14dc198530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 -- 172.21.15.73:0/3390139569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc101680 msgr2=0x7f14dc198a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f14dc101680 0x7f14dc198a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14dc19d540 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14db59e700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 0x7f14dc19ce00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007d70 tx=0x7f14d400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d4016020 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14dc19d830 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f14d400ad40 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f14d40115c0 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.473+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14dc1a50b0 con 0x7f14dc106d20 2024-03-09T00:05:04.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.474+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f14dc104da0 con 0x7f14dc106d20 2024-03-09T00:05:04.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.475+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f14d4019020 con 0x7f14dc106d20 2024-03-09T00:05:04.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14d3fff700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f14c80a2920 0x7f14c80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:04.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14dad9d700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f14c80a2920 0x7f14c80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:04.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f14d40cf340 con 0x7f14dc106d20 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14d3fff700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] conn(0x7f14c80ada80 0x7f14c80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f14d4018e70 con 0x7f14c80ada80 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=90) v2 ==== 24+0+0 (secure 0 0 0) 0x7f14d40cf790 con 0x7f14dc106d20 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14da59c700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] conn(0x7f14c80ada80 0x7f14c80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.476+0000 7f14dad9d700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f14c80a2920 0x7f14c80a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f14cc005fd0 tx=0x7f14cc005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.477+0000 7f14da59c700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] conn(0x7f14c80ada80 0x7f14c80aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:04.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.477+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== osd.6 v2:172.21.15.73:6818/1949163006 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f14d4018e70 con 0x7f14c80ada80 2024-03-09T00:05:04.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.493+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 --> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f14dc1023d0 con 0x7f14c80ada80 2024-03-09T00:05:04.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14d3fff700 1 -- 172.21.15.73:0/3390139569 <== osd.6 v2:172.21.15.73:6818/1949163006 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f14dc1023d0 con 0x7f14c80ada80 2024-03-09T00:05:04.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] conn(0x7f14c80ada80 msgr2=0x7f14c80aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] conn(0x7f14c80ada80 0x7f14c80aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f14c80a2920 msgr2=0x7f14c80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f14c80a2920 0x7f14c80a4de0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f14cc005fd0 tx=0x7f14cc005db0 comp rx=0 tx=0).stop 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 msgr2=0x7f14dc19ce00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 --2- 172.21.15.73:0/3390139569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f14dc106d20 0x7f14dc19ce00 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7f14d4007d70 tx=0x7f14d400a340 comp rx=0 tx=0).stop 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14db59e700 1 -- 172.21.15.73:0/3390139569 reap_dead start 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 shutdown_connections 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 >> 172.21.15.73:0/3390139569 conn(0x7f14dc077c10 msgr2=0x7f14dc0fff90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 shutdown_connections 2024-03-09T00:05:04.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:04.494+0000 7f14e1055700 1 -- 172.21.15.73:0/3390139569 wait complete. 2024-03-09T00:05:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]: dispatch 2024-03-09T00:05:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:04 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]: dispatch 2024-03-09T00:05:05.273 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-09T00:05:05.274 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:05 smithi073 ceph-mon[32309]: osdmap e90: 12 total, 12 up, 12 in 2024-03-09T00:05:05.301 INFO:teuthology.orchestra.run.smithi073.stdout:197568495666 2024-03-09T00:05:05.302 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-09T00:05:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-03-09T00:05:05.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-09T00:05:05.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-03-09T00:05:05.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-09T00:05:05.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-09T00:05:05.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-09T00:05:05.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:05 smithi179 ceph-mon[36637]: osdmap e90: 12 total, 12 up, 12 in 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: pgmap v319: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [2, 4]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.2", "id": [3, 9]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.e", "id": [5, 9]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.1b", "id": [4, 6]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.3a", "id": [4, 8]}]': finished 2024-03-09T00:05:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.7", "id": [5, 8]}]': finished 2024-03-09T00:05:05.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.10", "id": [9, 8]}]': finished 2024-03-09T00:05:05.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.12", "id": [9, 10]}]': finished 2024-03-09T00:05:05.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "4.19", "id": [9, 8]}]': finished 2024-03-09T00:05:05.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:05 smithi102 ceph-mon[36106]: osdmap e90: 12 total, 12 up, 12 in 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 -- 172.21.15.73:0/461061414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac100fc0 msgr2=0x7f50ac101440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 --2- 172.21.15.73:0/461061414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac100fc0 0x7f50ac101440 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f509c009990 tx=0x7f509c009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 -- 172.21.15.73:0/461061414 shutdown_connections 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 --2- 172.21.15.73:0/461061414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50ac101980 0x7f50ac10db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 --2- 172.21.15.73:0/461061414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50ac105ce0 0x7f50ac1060c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 --2- 172.21.15.73:0/461061414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac100fc0 0x7f50ac101440 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 -- 172.21.15.73:0/461061414 >> 172.21.15.73:0/461061414 conn(0x7f50ac077ba0 msgr2=0x7f50ac077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 -- 172.21.15.73:0/461061414 shutdown_connections 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.743+0000 7f50b3a39700 1 -- 172.21.15.73:0/461061414 wait complete. 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 Processor -- start 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 -- start start 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50ac100fc0 0x7f50ac1983f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50ac101980 0x7f50ac198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 0x7f50ac19ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac111050 con 0x7f50ac105ce0 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac110ed0 con 0x7f50ac100fc0 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b3a39700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50ac1111d0 con 0x7f50ac101980 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 0x7f50ac19ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:05.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 0x7f50ac19ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:53554/0 (socket says 172.21.15.73:53554) 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 -- 172.21.15.73:0/975598430 learned_addr learned my addr 172.21.15.73:0/975598430 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b0fd4700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50ac101980 0x7f50ac198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b17d5700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50ac100fc0 0x7f50ac1983f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 -- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50ac101980 msgr2=0x7f50ac198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f50ac101980 0x7f50ac198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 -- 172.21.15.73:0/975598430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50ac100fc0 msgr2=0x7f50ac1983f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f50ac100fc0 0x7f50ac1983f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.744+0000 7f50b1fd6700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50ac19d400 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50b1fd6700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 0x7f50ac19ccc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50a4007d70 tx=0x7f50a400a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a4016020 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f50a400ad40 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50ac19d6f0 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f50a40116f0 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.745+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50ac1a4f70 con 0x7f50ac105ce0 2024-03-09T00:05:05.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.746+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f50ac062470 con 0x7f50ac105ce0 2024-03-09T00:05:05.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.747+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f50a4019020 con 0x7f50ac105ce0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.747+0000 7f50a27fc700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50980a2920 0x7f50980a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50b17d5700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50980a2920 0x7f50980a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f50a40cf4e0 con 0x7f50ac105ce0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50a27fc700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] conn(0x7f50980ada80 0x7f50980aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f50a400bf30 con 0x7f50980ada80 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f50a40cf930 con 0x7f50ac105ce0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50b17d5700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50980a2920 0x7f50980a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f50a8005ea0 tx=0x7f50a8005e30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.748+0000 7f50b0fd4700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] conn(0x7f50980ada80 0x7f50980aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.749+0000 7f50b0fd4700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] conn(0x7f50980ada80 0x7f50980aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:05.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.749+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== osd.8 v2:172.21.15.179:6816/224478801 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f50a400bf30 con 0x7f50980ada80 2024-03-09T00:05:05.758 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.758 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.758 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.759 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.761 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.761 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.764 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:05.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.777+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 --> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f50ac0628f0 con 0x7f50980ada80 2024-03-09T00:05:05.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.779+0000 7f50a27fc700 1 -- 172.21.15.73:0/975598430 <== osd.8 v2:172.21.15.179:6816/224478801 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f50ac0628f0 con 0x7f50980ada80 2024-03-09T00:05:05.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] conn(0x7f50980ada80 msgr2=0x7f50980aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] conn(0x7f50980ada80 0x7f50980aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50980a2920 msgr2=0x7f50980a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50980a2920 0x7f50980a4de0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f50a8005ea0 tx=0x7f50a8005e30 comp rx=0 tx=0).stop 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 msgr2=0x7f50ac19ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 --2- 172.21.15.73:0/975598430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f50ac105ce0 0x7f50ac19ccc0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f50a4007d70 tx=0x7f50a400a340 comp rx=0 tx=0).stop 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b1fd6700 1 -- 172.21.15.73:0/975598430 reap_dead start 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.783+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 shutdown_connections 2024-03-09T00:05:05.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.784+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 >> 172.21.15.73:0/975598430 conn(0x7f50ac077ba0 msgr2=0x7f50ac0ff7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:05.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.788+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 shutdown_connections 2024-03-09T00:05:05.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:05.788+0000 7f50b3a39700 1 -- 172.21.15.73:0/975598430 wait complete. 2024-03-09T00:05:06.298 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:06 smithi073 ceph-mon[32309]: osdmap e91: 12 total, 12 up, 12 in 2024-03-09T00:05:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:06 smithi179 ceph-mon[36637]: osdmap e91: 12 total, 12 up, 12 in 2024-03-09T00:05:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:06 smithi102 ceph-mon[36106]: osdmap e91: 12 total, 12 up, 12 in 2024-03-09T00:05:06.737 INFO:teuthology.orchestra.run.smithi073.stdout:253403070504 2024-03-09T00:05:06.738 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-09T00:05:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:07 smithi179 ceph-mon[36637]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:07.369 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:07 smithi073 ceph-mon[32309]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:07 smithi102 ceph-mon[36106]: pgmap v322: 129 pgs: 4 peering, 125 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:09.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:09 smithi073 ceph-mon[32309]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:09.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:09 smithi179 ceph-mon[36637]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:09.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:09 smithi102 ceph-mon[36106]: pgmap v323: 129 pgs: 3 peering, 126 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 639 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:11.317 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:11 smithi073 ceph-mon[32309]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:11.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:11 smithi179 ceph-mon[36637]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:11 smithi102 ceph-mon[36106]: pgmap v324: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:13.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:13 smithi073 ceph-mon[32309]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:13.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:13 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:05:13.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:13 smithi179 ceph-mon[36637]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:13.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:13 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:05:13.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:13 smithi102 ceph-mon[36106]: pgmap v325: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:13.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:13 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:05:13.689 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:13.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 -- 172.21.15.73:0/3398681587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4100f70 msgr2=0x7f28e4101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 --2- 172.21.15.73:0/3398681587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4100f70 0x7f28e4101350 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f28d80099c0 tx=0x7f28d8009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 -- 172.21.15.73:0/3398681587 shutdown_connections 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 --2- 172.21.15.73:0/3398681587 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4100f70 0x7f28e4101350 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 --2- 172.21.15.73:0/3398681587 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4105fc0 0x7f28e410db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 --2- 172.21.15.73:0/3398681587 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28e4101920 0x7f28e41058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 -- 172.21.15.73:0/3398681587 >> 172.21.15.73:0/3398681587 conn(0x7f28e4076d30 msgr2=0x7f28e4077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 -- 172.21.15.73:0/3398681587 shutdown_connections 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.904+0000 7f28eac1f700 1 -- 172.21.15.73:0/3398681587 wait complete. 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 Processor -- start 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 -- start start 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 0x7f28e41985c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28e4101920 0x7f28e4198b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 0x7f28e419ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:13.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4110fa0 con 0x7f28e4105fc0 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4110e20 con 0x7f28e4101920 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28eac1f700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28e4111120 con 0x7f28e4100f70 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e91bc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 0x7f28e419ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e89bb700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 0x7f28e41985c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e89bb700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 0x7f28e41985c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:37604/0 (socket says 172.21.15.73:37604) 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e91bc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 0x7f28e419ce90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47336/0 (socket says 172.21.15.73:47336) 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e89bb700 1 -- 172.21.15.73:0/1335573098 learned_addr learned my addr 172.21.15.73:0/1335573098 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.905+0000 7f28e3fff700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28e4101920 0x7f28e4198b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 -- 172.21.15.73:0/1335573098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 msgr2=0x7f28e41985c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 0x7f28e41985c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 -- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28e4101920 msgr2=0x7f28e4198b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f28e4101920 0x7f28e4198b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28e419d5d0 con 0x7f28e4105fc0 2024-03-09T00:05:13.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e89bb700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f28e4100f70 0x7f28e41985c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e91bc700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 0x7f28e419ce90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f28dc007f10 tx=0x7f28dc00d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc004db0 con 0x7f28e4105fc0 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f28dc009e20 con 0x7f28e4105fc0 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f28dc015400 con 0x7f28e4105fc0 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28e419d8c0 con 0x7f28e4105fc0 2024-03-09T00:05:13.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.906+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28e41a5140 con 0x7f28e4105fc0 2024-03-09T00:05:13.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.907+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f28e4104d90 con 0x7f28e4105fc0 2024-03-09T00:05:13.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.908+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f28dc00db20 con 0x7f28e4105fc0 2024-03-09T00:05:13.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e1ffb700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f28d40a2920 0x7f28d40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e89bb700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f28d40a2920 0x7f28d40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f28dc0cf300 con 0x7f28e4105fc0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e1ffb700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] conn(0x7f28d40ada80 0x7f28d40aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f28dc0053a0 con 0x7f28d40ada80 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f28dc0cf7e0 con 0x7f28e4105fc0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e3fff700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] conn(0x7f28d40ada80 0x7f28d40aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.909+0000 7f28e89bb700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f28d40a2920 0x7f28d40a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009750 tx=0x7f28d8005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.910+0000 7f28e3fff700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] conn(0x7f28d40ada80 0x7f28d40aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:13.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.910+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== osd.10 v2:172.21.15.102:6824/2526045067 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f28dc0053a0 con 0x7f28d40ada80 2024-03-09T00:05:13.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.929+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 --> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f28e41023c0 con 0x7f28d40ada80 2024-03-09T00:05:13.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.929+0000 7f28e1ffb700 1 -- 172.21.15.73:0/1335573098 <== osd.10 v2:172.21.15.102:6824/2526045067 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f28e41023c0 con 0x7f28d40ada80 2024-03-09T00:05:13.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] conn(0x7f28d40ada80 msgr2=0x7f28d40aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] conn(0x7f28d40ada80 0x7f28d40aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f28d40a2920 msgr2=0x7f28d40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f28d40a2920 0x7f28d40a4de0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f28d8009750 tx=0x7f28d8005960 comp rx=0 tx=0).stop 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 msgr2=0x7f28e419ce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 --2- 172.21.15.73:0/1335573098 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f28e4105fc0 0x7f28e419ce90 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f28dc007f10 tx=0x7f28dc00d3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28e91bc700 1 -- 172.21.15.73:0/1335573098 reap_dead start 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 shutdown_connections 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 >> 172.21.15.73:0/1335573098 conn(0x7f28e4076d30 msgr2=0x7f28e41004a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 shutdown_connections 2024-03-09T00:05:13.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:13.930+0000 7f28eac1f700 1 -- 172.21.15.73:0/1335573098 wait complete. 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.584+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/14917317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 msgr2=0x7f2c84107150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.584+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/14917317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84107150 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f2c74009990 tx=0x7f2c74009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/14917317 shutdown_connections 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/14917317 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84107150 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/14917317 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c84101680 0x7f2c8410db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/14917317 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c84100cc0 0x7f2c84101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/14917317 >> 172.21.15.73:0/14917317 conn(0x7f2c84077cf0 msgr2=0x7f2c84078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/14917317 shutdown_connections 2024-03-09T00:05:14.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/14917317 wait complete. 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 Processor -- start 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.585+0000 7f2c8c7bf700 1 -- start start 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c84100cc0 0x7f2c84107bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c84101680 0x7f2c84108100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84108640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84110fa0 con 0x7f2c84106d70 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84110e20 con 0x7f2c84101680 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c84111120 con 0x7f2c84100cc0 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84108640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84108640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47348/0 (socket says 172.21.15.73:47348) 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 -- 172.21.15.73:0/2925871588 learned_addr learned my addr 172.21.15.73:0/2925871588 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:14.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c89d5a700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c84101680 0x7f2c84108100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8a55b700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c84100cc0 0x7f2c84107bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 -- 172.21.15.73:0/2925871588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c84100cc0 msgr2=0x7f2c84107bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2c84100cc0 0x7f2c84107bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 -- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c84101680 msgr2=0x7f2c84108100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2c84101680 0x7f2c84108100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c841a6b30 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8ad5c700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84108640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00ef30 tx=0x7f2c7c00c3a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c0092e0 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2c7c004bb0 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c841a6dc0 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.586+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2c7c004280 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.587+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c841a72d0 con 0x7f2c84106d70 2024-03-09T00:05:14.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.587+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2c840ff5b0 con 0x7f2c84106d70 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f2c7c01b030 con 0x7f2c84106d70 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2c700a2920 0x7f2c700a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c8a55b700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2c700a2920 0x7f2c700a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f2c7c0cf2a0 con 0x7f2c84106d70 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] conn(0x7f2c700ada80 0x7f2c700aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f2c7c01fe50 con 0x7f2c700ada80 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.589+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2c7c0cf6f0 con 0x7f2c84106d70 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.590+0000 7f2c8a55b700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2c700a2920 0x7f2c700a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400b2e0 tx=0x7f2c74005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.590+0000 7f2c89d5a700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] conn(0x7f2c700ada80 0x7f2c700aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.590+0000 7f2c89d5a700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] conn(0x7f2c700ada80 0x7f2c700aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.591+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== osd.7 v2:172.21.15.102:6816/1887369896 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f2c7c01fe50 con 0x7f2c700ada80 2024-03-09T00:05:14.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.607+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 --> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2c841023b0 con 0x7f2c700ada80 2024-03-09T00:05:14.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c7b7fe700 1 -- 172.21.15.73:0/2925871588 <== osd.7 v2:172.21.15.102:6816/1887369896 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2c841023b0 con 0x7f2c700ada80 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] conn(0x7f2c700ada80 msgr2=0x7f2c700aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] conn(0x7f2c700ada80 0x7f2c700aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2c700a2920 msgr2=0x7f2c700a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2c700a2920 0x7f2c700a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f2c7400b2e0 tx=0x7f2c74005960 comp rx=0 tx=0).stop 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 msgr2=0x7f2c84108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 --2- 172.21.15.73:0/2925871588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2c84106d70 0x7f2c84108640 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f2c7c00ef30 tx=0x7f2c7c00c3a0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8ad5c700 1 -- 172.21.15.73:0/2925871588 reap_dead start 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 shutdown_connections 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.608+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 >> 172.21.15.73:0/2925871588 conn(0x7f2c84077cf0 msgr2=0x7f2c840febd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.609+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 shutdown_connections 2024-03-09T00:05:14.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.609+0000 7f2c8c7bf700 1 -- 172.21.15.73:0/2925871588 wait complete. 2024-03-09T00:05:14.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 -- 172.21.15.73:0/2156389862 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058101680 msgr2=0x7f805810db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 --2- 172.21.15.73:0/2156389862 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058101680 0x7f805810db60 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f805000b0a0 tx=0x7f805000b3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 -- 172.21.15.73:0/2156389862 shutdown_connections 2024-03-09T00:05:14.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 --2- 172.21.15.73:0/2156389862 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058101680 0x7f805810db60 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 --2- 172.21.15.73:0/2156389862 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058106d20 0x7f8058107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 --2- 172.21.15.73:0/2156389862 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8058100ce0 0x7f8058101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 -- 172.21.15.73:0/2156389862 >> 172.21.15.73:0/2156389862 conn(0x7f8058077c10 msgr2=0x7f8058078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:14.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 -- 172.21.15.73:0/2156389862 shutdown_connections 2024-03-09T00:05:14.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.813+0000 7f805e958700 1 -- 172.21.15.73:0/2156389862 wait complete. 2024-03-09T00:05:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 Processor -- start 2024-03-09T00:05:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 -- start start 2024-03-09T00:05:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 0x7f80581973b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058101680 0x7f80581978f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8058106d20 0x7f805819bc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058110fa0 con 0x7f8058100ce0 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058110e20 con 0x7f8058101680 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.814+0000 7f805e958700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8058111120 con 0x7f8058106d20 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 0x7f80581973b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f80577fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058101680 0x7f80581978f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 0x7f80581973b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47370/0 (socket says 172.21.15.73:47370) 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f80577fe700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058101680 0x7f80581978f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:38980/0 (socket says 172.21.15.73:38980) 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 -- 172.21.15.73:0/1505191082 learned_addr learned my addr 172.21.15.73:0/1505191082 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f805cef5700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8058106d20 0x7f805819bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 -- 172.21.15.73:0/1505191082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8058106d20 msgr2=0x7f805819bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8058106d20 0x7f805819bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 -- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058101680 msgr2=0x7f80581978f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8058101680 0x7f80581978f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f805819c3c0 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f8057fff700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 0x7f80581973b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9a0 tx=0x7f804800ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f804800ca10 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f804800cbb0 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f805819c6b0 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8048018680 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.815+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80581a3f30 con 0x7f8058100ce0 2024-03-09T00:05:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.816+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f80580624f0 con 0x7f8058100ce0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.817+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f8048010430 con 0x7f8058100ce0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80557fa700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f80400a29a0 0x7f80400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80577fe700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f80400a29a0 0x7f80400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f80480cfbd0 con 0x7f8058100ce0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80557fa700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] conn(0x7f80400adb00 0x7f80400affa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8048019ec0 con 0x7f80400adb00 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80480d0020 con 0x7f8058100ce0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f80577fe700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f80400a29a0 0x7f80400a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8058078210 tx=0x7f804c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.818+0000 7f805cef5700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] conn(0x7f80400adb00 0x7f80400affa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.819+0000 7f805cef5700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] conn(0x7f80400adb00 0x7f80400affa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.820+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== osd.1 v2:172.21.15.102:6800/491071057 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f8048019ec0 con 0x7f80400adb00 2024-03-09T00:05:14.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.840+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 --> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f80580628f0 con 0x7f80400adb00 2024-03-09T00:05:14.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f80557fa700 1 -- 172.21.15.73:0/1505191082 <== osd.1 v2:172.21.15.102:6800/491071057 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f80580628f0 con 0x7f80400adb00 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] conn(0x7f80400adb00 msgr2=0x7f80400affa0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] conn(0x7f80400adb00 0x7f80400affa0 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f80400a29a0 msgr2=0x7f80400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f80400a29a0 0x7f80400a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f8058078210 tx=0x7f804c009260 comp rx=0 tx=0).stop 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 msgr2=0x7f80581973b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 --2- 172.21.15.73:0/1505191082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8058100ce0 0x7f80581973b0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7f804800e9a0 tx=0x7f804800ecb0 comp rx=0 tx=0).stop 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805cef5700 1 -- 172.21.15.73:0/1505191082 reap_dead start 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 shutdown_connections 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.841+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 >> 172.21.15.73:0/1505191082 conn(0x7f8058077c10 msgr2=0x7f80580ff990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:14.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.842+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 shutdown_connections 2024-03-09T00:05:14.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:14.842+0000 7f805e958700 1 -- 172.21.15.73:0/1505191082 wait complete. 2024-03-09T00:05:15.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:15 smithi073 ceph-mon[32309]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:15.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:15 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.323 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:15 smithi102 ceph-mon[36106]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:15.323 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.323 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.323 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.324 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:15 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:15 smithi179 ceph-mon[36637]: pgmap v326: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 KiB/s rd, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:15 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:16.259 INFO:teuthology.orchestra.run.smithi073.stdout:219043332142 2024-03-09T00:05:16.260 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-09T00:05:17.308 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:17 smithi073 ceph-mon[32309]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:17.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:17 smithi179 ceph-mon[36637]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:17.334 INFO:teuthology.orchestra.run.smithi073.stdout:55834574925 2024-03-09T00:05:17.335 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-09T00:05:17.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:17 smithi102 ceph-mon[36106]: pgmap v327: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 941 B/s rd, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:05:17.805 INFO:teuthology.orchestra.run.smithi073.stdout:309237645343 2024-03-09T00:05:17.806 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-09T00:05:18.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 -- 172.21.15.73:0/2370760888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2978101740 msgr2=0x7f2978101b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 --2- 172.21.15.73:0/2370760888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2978101740 0x7f2978101b20 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f29740099c0 tx=0x7f2974009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 -- 172.21.15.73:0/2370760888 shutdown_connections 2024-03-09T00:05:18.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 --2- 172.21.15.73:0/2370760888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2978101740 0x7f2978101b20 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 --2- 172.21.15.73:0/2370760888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 0x7f297810dc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 --2- 172.21.15.73:0/2370760888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f29781020f0 0x7f2978105980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 -- 172.21.15.73:0/2370760888 >> 172.21.15.73:0/2370760888 conn(0x7f29780772e0 msgr2=0x7f29780fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 -- 172.21.15.73:0/2370760888 shutdown_connections 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.900+0000 7f297fdd8700 1 -- 172.21.15.73:0/2370760888 wait complete. 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 Processor -- start 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 -- start start 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2978101740 0x7f2978198610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29781020f0 0x7f2978198b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 0x7f297819cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978111060 con 0x7f29781020f0 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2978110ee0 con 0x7f2978106060 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.901+0000 7f297fdd8700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29781111e0 con 0x7f2978101740 2024-03-09T00:05:18.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297db74700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2978101740 0x7f2978198610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 0x7f297819cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297db74700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2978101740 0x7f2978198610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:58642/0 (socket says 172.21.15.73:58642) 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297db74700 1 -- 172.21.15.73:0/1148770462 learned_addr learned my addr 172.21.15.73:0/1148770462 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297d373700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29781020f0 0x7f2978198b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 -- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2978101740 msgr2=0x7f2978198610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2978101740 0x7f2978198610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 -- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29781020f0 msgr2=0x7f2978198b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f29781020f0 0x7f2978198b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.902+0000 7f297e375700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297819d620 con 0x7f2978106060 2024-03-09T00:05:18.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f297e375700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 0x7f297819cee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f297000efc0 tx=0x7f297000c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:18.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29700092e0 con 0x7f2978106060 2024-03-09T00:05:18.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f297819d8b0 con 0x7f2978106060 2024-03-09T00:05:18.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2970004bb0 con 0x7f2978106060 2024-03-09T00:05:18.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29781a5190 con 0x7f2978106060 2024-03-09T00:05:18.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.903+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2970004280 con 0x7f2978106060 2024-03-09T00:05:18.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.904+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f295c001030 con 0x7f2978106060 2024-03-09T00:05:18.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.905+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f297001b030 con 0x7f2978106060 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f296effd700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29640a2920 0x7f29640a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f297db74700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29640a2920 0x7f29640a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f29700cf660 con 0x7f2978106060 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f296effd700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] conn(0x7f29640ada80 0x7f29640aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f29640b05f0 con 0x7f29640ada80 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.906+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f29700cfa10 con 0x7f2978106060 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.907+0000 7f297db74700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29640a2920 0x7f29640a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2974009750 tx=0x7f2974005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:18.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.907+0000 7f297d373700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] conn(0x7f29640ada80 0x7f29640aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:18.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.907+0000 7f297d373700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] conn(0x7f29640ada80 0x7f29640aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:18.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.908+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== osd.2 v2:172.21.15.179:6800/2018456550 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f29640b05f0 con 0x7f29640ada80 2024-03-09T00:05:18.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.924+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 --> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f295c002e60 con 0x7f29640ada80 2024-03-09T00:05:18.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f296effd700 1 -- 172.21.15.73:0/1148770462 <== osd.2 v2:172.21.15.179:6800/2018456550 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f295c002e60 con 0x7f29640ada80 2024-03-09T00:05:18.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] conn(0x7f29640ada80 msgr2=0x7f29640aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] conn(0x7f29640ada80 0x7f29640aff20 crc :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29640a2920 msgr2=0x7f29640a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29640a2920 0x7f29640a4de0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f2974009750 tx=0x7f2974005960 comp rx=0 tx=0).stop 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 msgr2=0x7f297819cee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 --2- 172.21.15.73:0/1148770462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2978106060 0x7f297819cee0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f297000efc0 tx=0x7f297000c3a0 comp rx=0 tx=0).stop 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297e375700 1 -- 172.21.15.73:0/1148770462 reap_dead start 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 shutdown_connections 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 >> 172.21.15.73:0/1148770462 conn(0x7f29780772e0 msgr2=0x7f2978100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 shutdown_connections 2024-03-09T00:05:18.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:18.925+0000 7f297fdd8700 1 -- 172.21.15.73:0/1148770462 wait complete. 2024-03-09T00:05:19.117 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:19 smithi073 ceph-mon[32309]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:19 smithi179 ceph-mon[36637]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:19.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:19 smithi102 ceph-mon[36106]: pgmap v328: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/3240293084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406a760 msgr2=0x7f9a7406ab40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/3240293084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406a760 0x7f9a7406ab40 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f9a640099c0 tx=0x7f9a64009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/3240293084 shutdown_connections 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/3240293084 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406a760 0x7f9a7406ab40 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/3240293084 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a74107120 0x7f9a7410ecf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/3240293084 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a7406b110 0x7f9a74106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/3240293084 >> 172.21.15.73:0/3240293084 conn(0x7f9a74077ba0 msgr2=0x7f9a74077fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/3240293084 shutdown_connections 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.122+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/3240293084 wait complete. 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 Processor -- start 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 -- start start 2024-03-09T00:05:19.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 0x7f9a74101090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 0x7f9a74101b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a74112120 con 0x7f9a7406b110 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a74111fa0 con 0x7f9a7406a760 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a7b8a0700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a741122a0 con 0x7f9a74107120 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a78e3b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a78e3b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32816/0 (socket says 172.21.15.73:32816) 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.123+0000 7f9a78e3b700 1 -- 172.21.15.73:0/2547140327 learned_addr learned my addr 172.21.15.73:0/2547140327 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a79e3d700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 0x7f9a74101b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a7963c700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 0x7f9a74101090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 -- 172.21.15.73:0/2547140327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 msgr2=0x7f9a74101b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 0x7f9a74101b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 -- 172.21.15.73:0/2547140327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 msgr2=0x7f9a74101090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:19.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 0x7f9a74101090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 -- 172.21.15.73:0/2547140327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a741a6a60 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a79e3d700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 0x7f9a74101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a7963c700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 0x7f9a74101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a78e3b700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000d950 tx=0x7f9a7000dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a70017070 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9a70005020 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a741a6d50 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9a7000fbc0 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.124+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a741a71e0 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.125+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a74068aa0 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.127+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9a7000fd60 con 0x7f9a7406b110 2024-03-09T00:05:19.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.127+0000 7f9a6a7fc700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 0x7f9a600a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.128+0000 7f9a7963c700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 0x7f9a600a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.128+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9a700ceaa0 con 0x7f9a7406b110 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.128+0000 7f9a7963c700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 0x7f9a600a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9a64000c00 tx=0x7f9a64005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.131+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9a7009b680 con 0x7f9a7406b110 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.343+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7f9a7402da90 con 0x7f9a7406b110 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.343+0000 7f9a6a7fc700 1 -- 172.21.15.73:0/2547140327 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9a7009af80 con 0x7f9a7406b110 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 msgr2=0x7f9a600a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 0x7f9a600a4de0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f9a64000c00 tx=0x7f9a64005fb0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 msgr2=0x7f9a741015d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f9a7000d950 tx=0x7f9a7000dc60 comp rx=0 tx=0).stop 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 shutdown_connections 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9a600a2920 0x7f9a600a4de0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9a74107120 0x7f9a74101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9a7406a760 0x7f9a74101090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 --2- 172.21.15.73:0/2547140327 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9a7406b110 0x7f9a741015d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 >> 172.21.15.73:0/2547140327 conn(0x7f9a74077ba0 msgr2=0x7f9a74100050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 shutdown_connections 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.345+0000 7f9a7b8a0700 1 -- 172.21.15.73:0/2547140327 wait complete. 2024-03-09T00:05:19.899 INFO:teuthology.orchestra.run.smithi073.stdout:163208757305 2024-03-09T00:05:20.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:20 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2547140327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-09T00:05:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:20 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2547140327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-09T00:05:20.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:20 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2547140327' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 -- 172.21.15.73:0/2248584420 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730069a90 msgr2=0x7f2730069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 --2- 172.21.15.73:0/2248584420 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730069a90 0x7f2730069ef0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f27200099c0 tx=0x7f2720009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 -- 172.21.15.73:0/2248584420 shutdown_connections 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 --2- 172.21.15.73:0/2248584420 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f273006a430 0x7f273010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 --2- 172.21.15.73:0/2248584420 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2730107e30 0x7f2730108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 --2- 172.21.15.73:0/2248584420 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730069a90 0x7f2730069ef0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 -- 172.21.15.73:0/2248584420 >> 172.21.15.73:0/2248584420 conn(0x7f2730077c10 msgr2=0x7f2730078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 -- 172.21.15.73:0/2248584420 shutdown_connections 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 -- 172.21.15.73:0/2248584420 wait complete. 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.793+0000 7f2735e98700 1 Processor -- start 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 -- start start 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2730069a90 0x7f27301985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f273006a430 0x7f2730198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 0x7f273019ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730112090 con 0x7f2730107e30 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730111f10 con 0x7f273006a430 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f2735e98700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2730112210 con 0x7f2730069a90 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 0x7f273019ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 0x7f273019ce70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32838/0 (socket says 172.21.15.73:32838) 2024-03-09T00:05:20.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 -- 172.21.15.73:0/3962479649 learned_addr learned my addr 172.21.15.73:0/3962479649 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272effd700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f273006a430 0x7f2730198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272f7fe700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2730069a90 0x7f27301985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 -- 172.21.15.73:0/3962479649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2730069a90 msgr2=0x7f27301985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2730069a90 0x7f27301985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 -- 172.21.15.73:0/3962479649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f273006a430 msgr2=0x7f2730198ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f273006a430 0x7f2730198ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f273019d5b0 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.794+0000 7f272ffff700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 0x7f273019ce70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2728007ee0 tx=0x7f272800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f272800dbd0 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f273019d8a0 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f272800dd70 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2728005a60 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27301a5120 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.795+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f27300624f0 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.797+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f272801a020 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.797+0000 7f272cff9700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27180a2920 0x7f27180a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.797+0000 7f272f7fe700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27180a2920 0x7f27180a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f27280cece0 con 0x7f2730107e30 2024-03-09T00:05:20.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272cff9700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] conn(0x7f27180ada80 0x7f27180aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f27180b05f0 con 0x7f27180ada80 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27280cf130 con 0x7f2730107e30 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272effd700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] conn(0x7f27180ada80 0x7f27180aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272f7fe700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27180a2920 0x7f27180a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f272400bbb0 tx=0x7f272400b590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.798+0000 7f272effd700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] conn(0x7f27180ada80 0x7f27180aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.799+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== osd.3 v2:172.21.15.73:6810/3007966557 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f27180b05f0 con 0x7f27180ada80 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.815+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 --> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f2730062920 con 0x7f27180ada80 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f272cff9700 1 -- 172.21.15.73:0/3962479649 <== osd.3 v2:172.21.15.73:6810/3007966557 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f2730062920 con 0x7f27180ada80 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] conn(0x7f27180ada80 msgr2=0x7f27180aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] conn(0x7f27180ada80 0x7f27180aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27180a2920 msgr2=0x7f27180a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27180a2920 0x7f27180a4de0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f272400bbb0 tx=0x7f272400b590 comp rx=0 tx=0).stop 2024-03-09T00:05:20.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 msgr2=0x7f273019ce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 --2- 172.21.15.73:0/3962479649 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2730107e30 0x7f273019ce70 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f2728007ee0 tx=0x7f272800d3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f272ffff700 1 -- 172.21.15.73:0/3962479649 reap_dead start 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 shutdown_connections 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 >> 172.21.15.73:0/3962479649 conn(0x7f2730077c10 msgr2=0x7f27300ff880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 shutdown_connections 2024-03-09T00:05:20.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:19.816+0000 7f2735e98700 1 -- 172.21.15.73:0/3962479649 wait complete. 2024-03-09T00:05:21.330 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:21 smithi073 ceph-mon[32309]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:21.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:21 smithi179 ceph-mon[36637]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:21.363 INFO:teuthology.orchestra.run.smithi073.stdout:107374182467 2024-03-09T00:05:21.364 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-09T00:05:21.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:21 smithi102 ceph-mon[36106]: pgmap v329: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:22.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 -- 172.21.15.73:0/2084819658 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 msgr2=0x7ff5cc104c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2084819658 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc104c70 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7ff5c400b0a0 tx=0x7ff5c400b3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 -- 172.21.15.73:0/2084819658 shutdown_connections 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2084819658 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc104c70 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2084819658 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5cc106cd0 0x7ff5cc069a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2084819658 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5cc069fd0 0x7ff5cc06a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 -- 172.21.15.73:0/2084819658 >> 172.21.15.73:0/2084819658 conn(0x7ff5cc077c10 msgr2=0x7ff5cc078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 -- 172.21.15.73:0/2084819658 shutdown_connections 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 -- 172.21.15.73:0/2084819658 wait complete. 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.153+0000 7ff5d353f700 1 Processor -- start 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 -- start start 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5cc069fd0 0x7ff5cc1a46f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc1a4c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5cc106cd0 0x7ff5cc19e8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:22.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5cc118690 con 0x7ff5cc1047e0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5cc118510 con 0x7ff5cc106cd0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d353f700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5cc118810 con 0x7ff5cc069fd0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc1a4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc1a4c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32864/0 (socket says 172.21.15.73:32864) 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 -- 172.21.15.73:0/2292544350 learned_addr learned my addr 172.21.15.73:0/2292544350 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d1adc700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5cc106cd0 0x7ff5cc19e8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d12db700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5cc069fd0 0x7ff5cc1a46f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 -- 172.21.15.73:0/2292544350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5cc069fd0 msgr2=0x7ff5cc1a46f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5cc069fd0 0x7ff5cc1a46f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 -- 172.21.15.73:0/2292544350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5cc106cd0 msgr2=0x7ff5cc19e8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5cc106cd0 0x7ff5cc19e8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5cc19f010 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.154+0000 7ff5d0ada700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc1a4c30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00d980 tx=0x7ff5bc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc017070 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5bc005020 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5bc00fbf0 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5cc19f300 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5cc1ab5c0 con 0x7ff5cc1047e0 2024-03-09T00:05:22.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.155+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7ff5cc1041a0 con 0x7ff5cc1047e0 2024-03-09T00:05:22.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.157+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7ff5bc0051d0 con 0x7ff5cc1047e0 2024-03-09T00:05:22.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.157+0000 7ff5c27fc700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5b80a2920 0x7ff5b80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:22.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.157+0000 7ff5d12db700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5b80a2920 0x7ff5b80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.157+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7ff5bc0cf330 con 0x7ff5cc1047e0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5c27fc700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] conn(0x7ff5b80ada80 0x7ff5b80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7ff5b80b05f0 con 0x7ff5b80ada80 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5bc0cf780 con 0x7ff5cc1047e0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5d1adc700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] conn(0x7ff5b80ada80 0x7ff5b80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5d12db700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5b80a2920 0x7ff5b80a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc19f9f0 tx=0x7ff5c800a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.158+0000 7ff5d1adc700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] conn(0x7ff5b80ada80 0x7ff5b80aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:22.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.159+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== osd.9 v2:172.21.15.73:6826/1773882248 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7ff5b80b05f0 con 0x7ff5b80ada80 2024-03-09T00:05:22.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 --> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7ff5cc1012c0 con 0x7ff5b80ada80 2024-03-09T00:05:22.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5c27fc700 1 -- 172.21.15.73:0/2292544350 <== osd.9 v2:172.21.15.73:6826/1773882248 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7ff5cc1012c0 con 0x7ff5b80ada80 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] conn(0x7ff5b80ada80 msgr2=0x7ff5b80aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] conn(0x7ff5b80ada80 0x7ff5b80aff20 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5b80a2920 msgr2=0x7ff5b80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5b80a2920 0x7ff5b80a4de0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc19f9f0 tx=0x7ff5c800a380 comp rx=0 tx=0).stop 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 msgr2=0x7ff5cc1a4c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 --2- 172.21.15.73:0/2292544350 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5cc1047e0 0x7ff5cc1a4c30 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7ff5bc00d980 tx=0x7ff5bc00dc90 comp rx=0 tx=0).stop 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d1adc700 1 -- 172.21.15.73:0/2292544350 reap_dead start 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 shutdown_connections 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 >> 172.21.15.73:0/2292544350 conn(0x7ff5cc077c10 msgr2=0x7ff5cc1037c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:22.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 shutdown_connections 2024-03-09T00:05:22.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:22.175+0000 7ff5d353f700 1 -- 172.21.15.73:0/2292544350 wait complete. 2024-03-09T00:05:23.036 INFO:tasks.cephadm.ceph_manager.ceph:need seq 163208757302 got 163208757305 for osd.5 2024-03-09T00:05:23.036 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/3322268331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 msgr2=0x7fcb58100ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/3322268331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58100ba0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fcb44009990 tx=0x7fcb44009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/3322268331 shutdown_connections 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/3322268331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb581010e0 0x7fcb5810d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/3322268331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 0x7fcb58106970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/3322268331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58100ba0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/3322268331 >> 172.21.15.73:0/3322268331 conn(0x7fcb58076040 msgr2=0x7fcb58076450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/3322268331 shutdown_connections 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.259+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/3322268331 wait complete. 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 Processor -- start 2024-03-09T00:05:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 -- start start 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58198620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb581010e0 0x7fcb58198b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 0x7fcb5819cef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810ff00 con 0x7fcb58100720 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb5810fd80 con 0x7fcb58106590 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5ea7b700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb58110080 con 0x7fcb581010e0 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58198620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58198620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32880/0 (socket says 172.21.15.73:32880) 2024-03-09T00:05:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 -- 172.21.15.73:0/2330311361 learned_addr learned my addr 172.21.15.73:0/2330311361 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb57fff700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb581010e0 0x7fcb58198b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5d018700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 0x7fcb5819cef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 -- 172.21.15.73:0/2330311361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb581010e0 msgr2=0x7fcb58198b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcb581010e0 0x7fcb58198b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 -- 172.21.15.73:0/2330311361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 msgr2=0x7fcb5819cef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 0x7fcb5819cef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb5819d630 con 0x7fcb58100720 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5d018700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcb58106590 0x7fcb5819cef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.260+0000 7fcb5c817700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58198620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00ddc0 tx=0x7fcb4c00a530 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.261+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c01b070 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.261+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcb4c00ad40 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.261+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb5819d920 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.261+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcb4c0167d0 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.261+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb581a51a0 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.262+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb580624b0 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.263+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fcb4c01e070 con 0x7fcb58100720 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.263+0000 7fcb55ffb700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fcb480a2920 0x7fcb480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.263+0000 7fcb57fff700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fcb480a2920 0x7fcb480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.263+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fcb4c0ce9d0 con 0x7fcb58100720 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb55ffb700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] conn(0x7fcb480ada80 0x7fcb480aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb4c001d30 con 0x7fcb480ada80 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb4c0d32c0 con 0x7fcb58100720 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb5d018700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] conn(0x7fcb480ada80 0x7fcb480aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb57fff700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fcb480a2920 0x7fcb480a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098e0 tx=0x7fcb44005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.264+0000 7fcb5d018700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] conn(0x7fcb480ada80 0x7fcb480aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.265+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== osd.0 v2:172.21.15.73:6802/3305091634 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcb4c001d30 con 0x7fcb480ada80 2024-03-09T00:05:23.269 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:23 smithi073 ceph-mon[32309]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:23.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.288+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 --> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb58062960 con 0x7fcb480ada80 2024-03-09T00:05:23.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.288+0000 7fcb55ffb700 1 -- 172.21.15.73:0/2330311361 <== osd.0 v2:172.21.15.73:6802/3305091634 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fcb58062960 con 0x7fcb480ada80 2024-03-09T00:05:23.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.288+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] conn(0x7fcb480ada80 msgr2=0x7fcb480aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.288+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] conn(0x7fcb480ada80 0x7fcb480aff20 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:23.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fcb480a2920 msgr2=0x7fcb480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fcb480a2920 0x7fcb480a4de0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fcb440098e0 tx=0x7fcb44005960 comp rx=0 tx=0).stop 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 msgr2=0x7fcb58198620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 --2- 172.21.15.73:0/2330311361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcb58100720 0x7fcb58198620 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fcb4c00ddc0 tx=0x7fcb4c00a530 comp rx=0 tx=0).stop 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5d018700 1 -- 172.21.15.73:0/2330311361 reap_dead start 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 shutdown_connections 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 >> 172.21.15.73:0/2330311361 conn(0x7fcb58076040 msgr2=0x7fcb580fe770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 shutdown_connections 2024-03-09T00:05:23.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:23.289+0000 7fcb5ea7b700 1 -- 172.21.15.73:0/2330311361 wait complete. 2024-03-09T00:05:23.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:23 smithi179 ceph-mon[36637]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:23.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:23 smithi102 ceph-mon[36106]: pgmap v330: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:23.734 INFO:teuthology.orchestra.run.smithi073.stdout:77309411401 2024-03-09T00:05:23.735 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-09T00:05:24.372 INFO:teuthology.orchestra.run.smithi073.stdout:287762808870 2024-03-09T00:05:24.372 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-09T00:05:24.420 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:24.421 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:25.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:25 smithi179 ceph-mon[36637]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:25.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:25 smithi073 ceph-mon[32309]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:25.422 INFO:teuthology.orchestra.run.smithi073.stdout:34359738451 2024-03-09T00:05:25.423 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-09T00:05:25.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:25 smithi102 ceph-mon[36106]: pgmap v331: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:25.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 -- 172.21.15.73:0/3300727273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc107170 msgr2=0x7f1fcc10ecf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/3300727273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc107170 0x7f1fcc10ecf0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f1fc400b0d0 tx=0x7f1fc400b3e0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 -- 172.21.15.73:0/3300727273 shutdown_connections 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/3300727273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc107170 0x7f1fcc10ecf0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/3300727273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fcc1020c0 0x7f1fcc1024a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/3300727273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fcc102a70 0x7f1fcc106a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 -- 172.21.15.73:0/3300727273 >> 172.21.15.73:0/3300727273 conn(0x7f1fcc078f00 msgr2=0x7f1fcc079310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 -- 172.21.15.73:0/3300727273 shutdown_connections 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.674+0000 7f1fd38f1700 1 -- 172.21.15.73:0/3300727273 wait complete. 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 Processor -- start 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 -- start start 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fcc1020c0 0x7f1fcc0740e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 0x7f1fcc074620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fcc107170 0x7f1fcc074b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:25.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc112130 con 0x7f1fcc102a70 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc111fb0 con 0x7f1fcc107170 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd38f1700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fcc1122b0 con 0x7f1fcc1020c0 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd0e8c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 0x7f1fcc074620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd0e8c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 0x7f1fcc074620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32888/0 (socket says 172.21.15.73:32888) 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.675+0000 7f1fd0e8c700 1 -- 172.21.15.73:0/2178888218 learned_addr learned my addr 172.21.15.73:0/2178888218 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd168d700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fcc1020c0 0x7f1fcc0740e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd1e8e700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fcc107170 0x7f1fcc074b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 -- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fcc1020c0 msgr2=0x7f1fcc0740e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fcc1020c0 0x7f1fcc0740e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 -- 172.21.15.73:0/2178888218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fcc107170 msgr2=0x7f1fcc074b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fcc107170 0x7f1fcc074b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fcc1a6a30 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd0e8c700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 0x7f1fcc074620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00d950 tx=0x7f1fbc00dc60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc017070 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fcc1a6d20 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fbc00fd70 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fbc00e3b0 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.676+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fcc1a7200 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.677+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1fcc0624b0 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.678+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f1fbc01a070 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fc27fc700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb80a2920 0x7f1fb80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fd168d700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb80a2920 0x7f1fb80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f1fbc0ceab0 con 0x7f1fcc102a70 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fc27fc700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] conn(0x7f1fb80ada80 0x7f1fb80aff20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:25.683 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1fbc009e50 con 0x7f1fb80ada80 2024-03-09T00:05:25.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_get_version_reply(handle=1 version=91) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1fbc0cef00 con 0x7f1fcc102a70 2024-03-09T00:05:25.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fd168d700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb80a2920 0x7f1fb80a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004630 tx=0x7f1fc800b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:25.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.679+0000 7f1fd1e8e700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] conn(0x7f1fb80ada80 0x7f1fb80aff20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:25.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.680+0000 7f1fd1e8e700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] conn(0x7f1fb80ada80 0x7f1fb80aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:25.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.681+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== osd.11 v2:172.21.15.179:6824/897506355 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1fbc009e50 con 0x7f1fb80ada80 2024-03-09T00:05:25.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.697+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 --> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1fcc0628b0 con 0x7f1fb80ada80 2024-03-09T00:05:25.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fc27fc700 1 -- 172.21.15.73:0/2178888218 <== osd.11 v2:172.21.15.179:6824/897506355 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f1fcc0628b0 con 0x7f1fb80ada80 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] conn(0x7f1fb80ada80 msgr2=0x7f1fb80aff20 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] conn(0x7f1fb80ada80 0x7f1fb80aff20 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb80a2920 msgr2=0x7f1fb80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb80a2920 0x7f1fb80a4de0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f1fc8004630 tx=0x7f1fc800b3c0 comp rx=0 tx=0).stop 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 msgr2=0x7f1fcc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 --2- 172.21.15.73:0/2178888218 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fcc102a70 0x7f1fcc074620 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f1fbc00d950 tx=0x7f1fbc00dc60 comp rx=0 tx=0).stop 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd1e8e700 1 -- 172.21.15.73:0/2178888218 reap_dead start 2024-03-09T00:05:25.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 shutdown_connections 2024-03-09T00:05:25.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 >> 172.21.15.73:0/2178888218 conn(0x7f1fcc078f00 msgr2=0x7f1fcc0ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:25.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.698+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 shutdown_connections 2024-03-09T00:05:25.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:25.699+0000 7f1fd38f1700 1 -- 172.21.15.73:0/2178888218 wait complete. 2024-03-09T00:05:27.239 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:27 smithi073 ceph-mon[32309]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:27.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:27 smithi179 ceph-mon[36637]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:27.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:27 smithi102 ceph-mon[36106]: pgmap v332: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:27.977 INFO:teuthology.orchestra.run.smithi073.stdout:343597383708 2024-03-09T00:05:27.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-09T00:05:28.268 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:28.268 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:28.269 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:28.269 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:28.273 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:28.273 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:29.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:29 smithi073 ceph-mon[32309]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:29.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:29 smithi102 ceph-mon[36106]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:29.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:29 smithi179 ceph-mon[36637]: pgmap v333: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 -- 172.21.15.73:0/3620849370 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c105fc0 msgr2=0x7f671c10db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 --2- 172.21.15.73:0/3620849370 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c105fc0 0x7f671c10db90 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f671400b0d0 tx=0x7f671400b3e0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 -- 172.21.15.73:0/3620849370 shutdown_connections 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 --2- 172.21.15.73:0/3620849370 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c105fc0 0x7f671c10db90 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 --2- 172.21.15.73:0/3620849370 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c100f70 0x7f671c101350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 --2- 172.21.15.73:0/3620849370 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c101920 0x7f671c1058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 -- 172.21.15.73:0/3620849370 >> 172.21.15.73:0/3620849370 conn(0x7f671c076d30 msgr2=0x7f671c077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 -- 172.21.15.73:0/3620849370 shutdown_connections 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.692+0000 7f6723d41700 1 -- 172.21.15.73:0/3620849370 wait complete. 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 Processor -- start 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 -- start start 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 0x7f671c197460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 0x7f671c19bd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c111050 con 0x7f671c101920 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c110ed0 con 0x7f671c105fc0 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6723d41700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f671c1111d0 con 0x7f671c100f70 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45962/0 (socket says 172.21.15.73:45962) 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 -- 172.21.15.73:0/3230788752 learned_addr learned my addr 172.21.15.73:0/3230788752 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:30.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67222de700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 0x7f671c19bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6721add700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 0x7f671c197460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 -- 172.21.15.73:0/3230788752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 msgr2=0x7f671c197460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 0x7f671c197460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 -- 172.21.15.73:0/3230788752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 msgr2=0x7f671c19bd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 0x7f671c19bd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67212dc700 1 -- 172.21.15.73:0/3230788752 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f671c19c470 con 0x7f671c101920 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f67222de700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 0x7f671c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.693+0000 7f6721add700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 0x7f671c197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f67212dc700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f670c00d950 tx=0x7f670c00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c017070 con 0x7f671c101920 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f670c00de30 con 0x7f671c101920 2024-03-09T00:05:30.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f671c19c760 con 0x7f671c101920 2024-03-09T00:05:30.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f670c0054c0 con 0x7f671c101920 2024-03-09T00:05:30.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.694+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f671c1a3fe0 con 0x7f671c101920 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.695+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f671c068aa0 con 0x7f671c101920 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.696+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f670c004750 con 0x7f671c101920 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.696+0000 7f6712ffd700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 0x7f67080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.697+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f670c0ce830 con 0x7f671c101920 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.697+0000 7f6721add700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 0x7f67080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:30.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.698+0000 7f6721add700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 0x7f67080a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67180099d0 tx=0x7f6718008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:30.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.702+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f670c09b410 con 0x7f671c101920 2024-03-09T00:05:30.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.904+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f671c02d830 con 0x7f671c101920 2024-03-09T00:05:30.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.905+0000 7f6712ffd700 1 -- 172.21.15.73:0/3230788752 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f670c09ad10 con 0x7f671c101920 2024-03-09T00:05:30.900 INFO:teuthology.orchestra.run.smithi073.stdout:141733920833 2024-03-09T00:05:30.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 msgr2=0x7f67080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:30.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 0x7f67080a4e60 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f67180099d0 tx=0x7f6718008040 comp rx=0 tx=0).stop 2024-03-09T00:05:30.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 msgr2=0x7f671c1979a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:30.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f670c00d950 tx=0x7f670c00dd10 comp rx=0 tx=0).stop 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 shutdown_connections 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67080a29a0 0x7f67080a4e60 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f671c105fc0 0x7f671c19bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f671c100f70 0x7f671c197460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 --2- 172.21.15.73:0/3230788752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f671c101920 0x7f671c1979a0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 >> 172.21.15.73:0/3230788752 conn(0x7f671c076d30 msgr2=0x7f671c0fed20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 shutdown_connections 2024-03-09T00:05:30.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:30.907+0000 7f6723d41700 1 -- 172.21.15.73:0/3230788752 wait complete. 2024-03-09T00:05:31.379 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:31 smithi073 ceph-mon[32309]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:31.379 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3230788752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-09T00:05:31.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:31 smithi102 ceph-mon[36106]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:31.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3230788752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-09T00:05:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:31 smithi179 ceph-mon[36637]: pgmap v334: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3230788752' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-09T00:05:31.917 INFO:tasks.cephadm.ceph_manager.ceph:need seq 141733920828 got 141733920833 for osd.4 2024-03-09T00:05:31.917 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:32.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 -- 172.21.15.73:0/1606774313 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 msgr2=0x7f7430106a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 --2- 172.21.15.73:0/1606774313 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430106a60 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f741c0099c0 tx=0x7f741c009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 -- 172.21.15.73:0/1606774313 shutdown_connections 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 --2- 172.21.15.73:0/1606774313 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7430107140 0x7f743010ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 --2- 172.21.15.73:0/1606774313 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f743006a7a0 0x7f743006ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 --2- 172.21.15.73:0/1606774313 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430106a60 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 -- 172.21.15.73:0/1606774313 >> 172.21.15.73:0/1606774313 conn(0x7f7430077de0 msgr2=0x7f74300781f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 -- 172.21.15.73:0/1606774313 shutdown_connections 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.148+0000 7f7436d39700 1 -- 172.21.15.73:0/1606774313 wait complete. 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 Processor -- start 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 -- start start 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f743006a7a0 0x7f7430198590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:32.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:32.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 0x7f743019ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:32.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7430112120 con 0x7f743006b150 2024-03-09T00:05:32.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7430111fa0 con 0x7f7430107140 2024-03-09T00:05:32.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7436d39700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74301122a0 con 0x7f743006a7a0 2024-03-09T00:05:32.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45984/0 (socket says 172.21.15.73:45984) 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 -- 172.21.15.73:0/1960684672 learned_addr learned my addr 172.21.15.73:0/1960684672 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f7434ad5700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f743006a7a0 0x7f7430198590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f74352d6700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 0x7f743019ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 -- 172.21.15.73:0/1960684672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f743006a7a0 msgr2=0x7f7430198590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:32.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f743006a7a0 0x7f7430198590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 -- 172.21.15.73:0/1960684672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 msgr2=0x7f743019ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.149+0000 7f742ffff700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 0x7f743019ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f742ffff700 1 -- 172.21.15.73:0/1960684672 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f743019d5a0 con 0x7f743006b150 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f74352d6700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 0x7f743019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f742ffff700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f741c0059d0 tx=0x7f741c0038b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:32.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c027070 con 0x7f743006b150 2024-03-09T00:05:32.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f741c004050 con 0x7f743006b150 2024-03-09T00:05:32.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f743019d830 con 0x7f743006b150 2024-03-09T00:05:32.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741c022de0 con 0x7f743006b150 2024-03-09T00:05:32.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.150+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74301a5110 con 0x7f743006b150 2024-03-09T00:05:32.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.151+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7430103540 con 0x7f743006b150 2024-03-09T00:05:32.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.153+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f741c02a070 con 0x7f743006b150 2024-03-09T00:05:32.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.154+0000 7f742dffb700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 0x7f74200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:32.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.154+0000 7f7434ad5700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 0x7f74200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:32.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.155+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f741c0d0550 con 0x7f743006b150 2024-03-09T00:05:32.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.155+0000 7f7434ad5700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 0x7f74200a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7424005950 tx=0x7f74240058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:32.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.159+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f741c031020 con 0x7f743006b150 2024-03-09T00:05:32.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.362+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f743002d830 con 0x7f743006b150 2024-03-09T00:05:32.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.362+0000 7f742dffb700 1 -- 172.21.15.73:0/1960684672 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f741c013070 con 0x7f743006b150 2024-03-09T00:05:32.357 INFO:teuthology.orchestra.run.smithi073.stdout:107374182470 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 msgr2=0x7f74200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 0x7f74200a4de0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f7424005950 tx=0x7f74240058e0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 msgr2=0x7f7430198ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f741c0059d0 tx=0x7f741c0038b0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 shutdown_connections 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74200a2920 0x7f74200a4de0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7430107140 0x7f743019ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f743006a7a0 0x7f7430198590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 --2- 172.21.15.73:0/1960684672 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f743006b150 0x7f7430198ad0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:32.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 >> 172.21.15.73:0/1960684672 conn(0x7f7430077de0 msgr2=0x7f74300fec30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:32.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 shutdown_connections 2024-03-09T00:05:32.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:32.364+0000 7f7436d39700 1 -- 172.21.15.73:0/1960684672 wait complete. 2024-03-09T00:05:33.284 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:33 smithi073 ceph-mon[32309]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:33.285 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:33 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1960684672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-09T00:05:33.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:33 smithi102 ceph-mon[36106]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:33.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:33 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1960684672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-09T00:05:33.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:33 smithi179 ceph-mon[36637]: pgmap v335: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 851 B/s rd, 1 op/s 2024-03-09T00:05:33.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:33 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1960684672' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-09T00:05:34.125 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:34.127 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 -- 172.21.15.73:0/455943316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 msgr2=0x7f9bd8101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 --2- 172.21.15.73:0/455943316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd8101350 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400b0d0 tx=0x7f9bd400b3e0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 -- 172.21.15.73:0/455943316 shutdown_connections 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 --2- 172.21.15.73:0/455943316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd8101350 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 --2- 172.21.15.73:0/455943316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 0x7f9bd810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 --2- 172.21.15.73:0/455943316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd81058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 -- 172.21.15.73:0/455943316 >> 172.21.15.73:0/455943316 conn(0x7f9bd8076d30 msgr2=0x7f9bd8077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 -- 172.21.15.73:0/455943316 shutdown_connections 2024-03-09T00:05:34.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.463+0000 7f9bdf207700 1 -- 172.21.15.73:0/455943316 wait complete. 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 Processor -- start 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 -- start start 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd8199920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 0x7f9bd81a4460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8110fc0 con 0x7f9bd8100f70 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8110e40 con 0x7f9bd8105fc0 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdf207700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9bd8111140 con 0x7f9bd8101920 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46002/0 (socket says 172.21.15.73:46002) 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 -- 172.21.15.73:0/3077873492 learned_addr learned my addr 172.21.15.73:0/3077873492 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bcffff700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd8199920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdd7a4700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 0x7f9bd81a4460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:34.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 -- 172.21.15.73:0/3077873492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 msgr2=0x7f9bd8199920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd8199920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 -- 172.21.15.73:0/3077873492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 msgr2=0x7f9bd81a4460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 0x7f9bd81a4460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 -- 172.21.15.73:0/3077873492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9bd81a49a0 con 0x7f9bd8100f70 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bcffff700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd8199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.464+0000 7f9bdcfa3700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400b820 tx=0x7f9bd4004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.465+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4021020 con 0x7f9bd8100f70 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.465+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9bd81a4b70 con 0x7f9bd8100f70 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.465+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9bd4007d40 con 0x7f9bd8100f70 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.465+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9bd4005370 con 0x7f9bd8100f70 2024-03-09T00:05:34.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.465+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9bd81a5070 con 0x7f9bd8100f70 2024-03-09T00:05:34.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.466+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9bd8068aa0 con 0x7f9bd8100f70 2024-03-09T00:05:34.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.467+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9bd4024020 con 0x7f9bd8100f70 2024-03-09T00:05:34.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.468+0000 7f9bcdffb700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 0x7f9bc00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:34.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.468+0000 7f9bcffff700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 0x7f9bc00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:34.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.468+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9bd400f020 con 0x7f9bd8100f70 2024-03-09T00:05:34.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.468+0000 7f9bcffff700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 0x7f9bc00a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8100900 tx=0x7f9bc800b2c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:34.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.472+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9bd409cf40 con 0x7f9bd8100f70 2024-03-09T00:05:34.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.671+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f9bd802d950 con 0x7f9bd8100f70 2024-03-09T00:05:34.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.671+0000 7f9bcdffb700 1 -- 172.21.15.73:0/3077873492 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9bd409c840 con 0x7f9bd8100f70 2024-03-09T00:05:34.667 INFO:teuthology.orchestra.run.smithi073.stdout:197568495672 2024-03-09T00:05:34.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 msgr2=0x7f9bc00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 0x7f9bc00a4e60 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f9bd8100900 tx=0x7f9bc800b2c0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 msgr2=0x7f9bd81993e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f9bd400b820 tx=0x7f9bd4004e60 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 shutdown_connections 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9bd8100f70 0x7f9bd81993e0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9bc00a29a0 0x7f9bc00a4e60 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9bd8105fc0 0x7f9bd81a4460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.673+0000 7f9bdf207700 1 --2- 172.21.15.73:0/3077873492 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9bd8101920 0x7f9bd8199920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.674+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 >> 172.21.15.73:0/3077873492 conn(0x7f9bd8076d30 msgr2=0x7f9bd8105150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.674+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 shutdown_connections 2024-03-09T00:05:34.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:34.674+0000 7f9bdf207700 1 -- 172.21.15.73:0/3077873492 wait complete. 2024-03-09T00:05:35.353 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:35 smithi073 ceph-mon[32309]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:35.353 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3077873492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-09T00:05:35.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:35 smithi102 ceph-mon[36106]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:35.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3077873492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-09T00:05:35.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:35 smithi179 ceph-mon[36637]: pgmap v336: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:35.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3077873492' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.195+0000 7f766459a700 1 -- 172.21.15.73:0/3258020190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c101960 msgr2=0x7f765c1059b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.195+0000 7f766459a700 1 --2- 172.21.15.73:0/3258020190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c101960 0x7f765c1059b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f764c0099c0 tx=0x7f764c009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- 172.21.15.73:0/3258020190 shutdown_connections 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- 172.21.15.73:0/3258020190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c106090 0x7f765c10dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- 172.21.15.73:0/3258020190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 0x7f765c101390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- 172.21.15.73:0/3258020190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c101960 0x7f765c1059b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- 172.21.15.73:0/3258020190 >> 172.21.15.73:0/3258020190 conn(0x7f765c077e10 msgr2=0x7f765c078220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:36.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- 172.21.15.73:0/3258020190 shutdown_connections 2024-03-09T00:05:36.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- 172.21.15.73:0/3258020190 wait complete. 2024-03-09T00:05:36.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 Processor -- start 2024-03-09T00:05:36.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- start start 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 0x7f765c1994f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c101960 0x7f765c199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c111150 con 0x7f765c106090 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c110fd0 con 0x7f765c100fb0 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.196+0000 7f766459a700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765c1112d0 con 0x7f765c101960 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46022/0 (socket says 172.21.15.73:46022) 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 -- 172.21.15.73:0/1727433403 learned_addr learned my addr 172.21.15.73:0/1727433403 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662336700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 0x7f765c1994f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:36.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7661b35700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c101960 0x7f765c199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 -- 172.21.15.73:0/1727433403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c101960 msgr2=0x7f765c199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c101960 0x7f765c199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 -- 172.21.15.73:0/1727433403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 msgr2=0x7f765c1994f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 0x7f765c1994f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 -- 172.21.15.73:0/1727433403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765c1a4ab0 con 0x7f765c106090 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f7662b37700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f765400b970 tx=0x7f765400bd30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7654019070 con 0x7f765c106090 2024-03-09T00:05:36.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7654014440 con 0x7f765c106090 2024-03-09T00:05:36.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f765c1a4ce0 con 0x7f765c106090 2024-03-09T00:05:36.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76540075d0 con 0x7f765c106090 2024-03-09T00:05:36.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.197+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f765c1a51a0 con 0x7f765c106090 2024-03-09T00:05:36.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.198+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f765c068aa0 con 0x7f765c106090 2024-03-09T00:05:36.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.199+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f765401c070 con 0x7f765c106090 2024-03-09T00:05:36.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.200+0000 7f76537fe700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 0x7f76480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:36.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.200+0000 7f7662336700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 0x7f76480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:36.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.200+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f76540ce760 con 0x7f765c106090 2024-03-09T00:05:36.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.200+0000 7f7662336700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 0x7f76480a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7658006fd0 tx=0x7f7658008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:36.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.204+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f765409b450 con 0x7f765c106090 2024-03-09T00:05:36.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.405+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f765c02d830 con 0x7f765c106090 2024-03-09T00:05:36.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.406+0000 7f76537fe700 1 -- 172.21.15.73:0/1727433403 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f765409ad50 con 0x7f765c106090 2024-03-09T00:05:36.401 INFO:teuthology.orchestra.run.smithi073.stdout:77309411405 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.407+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 msgr2=0x7f76480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 0x7f76480a4de0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f7658006fd0 tx=0x7f7658008040 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 msgr2=0x7f765c1a4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f765400b970 tx=0x7f765400bd30 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 shutdown_connections 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f765c106090 0x7f765c1a4570 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f76480a2920 0x7f76480a4de0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f765c100fb0 0x7f765c1994f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 --2- 172.21.15.73:0/1727433403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f765c101960 0x7f765c199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:36.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 >> 172.21.15.73:0/1727433403 conn(0x7f765c077e10 msgr2=0x7f765c0fec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:36.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 shutdown_connections 2024-03-09T00:05:36.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:36.408+0000 7f766459a700 1 -- 172.21.15.73:0/1727433403 wait complete. 2024-03-09T00:05:36.922 INFO:tasks.cephadm.ceph_manager.ceph:need seq 107374182467 got 107374182470 for osd.3 2024-03-09T00:05:36.922 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:37.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:37 smithi073 ceph-mon[32309]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:37.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:37 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1727433403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-09T00:05:37.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:37 smithi102 ceph-mon[36106]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:37.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:37 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1727433403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-09T00:05:37.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:37 smithi179 ceph-mon[36637]: pgmap v337: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:37.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:37 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1727433403' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-09T00:05:39.397 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:39 smithi073 ceph-mon[32309]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:39.427 INFO:tasks.cephadm.ceph_manager.ceph:need seq 197568495666 got 197568495672 for osd.6 2024-03-09T00:05:39.427 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:39 smithi102 ceph-mon[36106]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:39.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:39 smithi179 ceph-mon[36637]: pgmap v338: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:40.231 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:40 smithi073 ceph-mon[32309]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:40.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:40 smithi102 ceph-mon[36106]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:40.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:40 smithi179 ceph-mon[36637]: pgmap v339: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:41.690 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411401 got 77309411405 for osd.2 2024-03-09T00:05:41.690 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 -- 172.21.15.73:0/2457835441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8100f70 msgr2=0x7f9ca8101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/2457835441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8100f70 0x7f9ca8101350 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f9ca40099c0 tx=0x7f9ca4009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 -- 172.21.15.73:0/2457835441 shutdown_connections 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/2457835441 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8100f70 0x7f9ca8101350 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/2457835441 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8105fc0 0x7f9ca810db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/2457835441 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8101920 0x7f9ca81058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 -- 172.21.15.73:0/2457835441 >> 172.21.15.73:0/2457835441 conn(0x7f9ca8076d30 msgr2=0x7f9ca8077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 -- 172.21.15.73:0/2457835441 shutdown_connections 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.355+0000 7f9cafe9d700 1 -- 172.21.15.73:0/2457835441 wait complete. 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 Processor -- start 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 -- start start 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8100f70 0x7f9ca81993b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 0x7f9ca81a4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8110fd0 con 0x7f9ca8101920 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8110e50 con 0x7f9ca8105fc0 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cafe9d700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8111150 con 0x7f9ca8100f70 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:42.365 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51282/0 (socket says 172.21.15.73:51282) 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 -- 172.21.15.73:0/69493889 learned_addr learned my addr 172.21.15.73:0/69493889 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cadc39700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8100f70 0x7f9ca81993b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cae43a700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 0x7f9ca81a4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 -- 172.21.15.73:0/69493889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8100f70 msgr2=0x7f9ca81993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8100f70 0x7f9ca81993b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 -- 172.21.15.73:0/69493889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 msgr2=0x7f9ca81a4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.356+0000 7f9cad438700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 0x7f9ca81a4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9cad438700 1 -- 172.21.15.73:0/69493889 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca81a4970 con 0x7f9ca8101920 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9cae43a700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 0x7f9ca81a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9cad438700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800c8f0 tx=0x7f9c9800cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98007940 con 0x7f9ca8101920 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca81a4ba0 con 0x7f9ca8101920 2024-03-09T00:05:42.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c9800f450 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c98018660 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.357+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca81a50e0 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.358+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca8068aa0 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.359+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f9c9800f600 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.360+0000 7f9c9effd700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 0x7f9c940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.360+0000 7f9cadc39700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 0x7f9c940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.360+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f9c980cedc0 con 0x7f9ca8101920 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.360+0000 7f9cadc39700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 0x7f9c940a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000c00 tx=0x7f9ca4005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:42.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.364+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9c98014030 con 0x7f9ca8101920 2024-03-09T00:05:42.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.566+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7f9ca802d9f0 con 0x7f9ca8101920 2024-03-09T00:05:42.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.566+0000 7f9c9effd700 1 -- 172.21.15.73:0/69493889 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f9c9809b9a0 con 0x7f9ca8101920 2024-03-09T00:05:42.562 INFO:teuthology.orchestra.run.smithi073.stdout:253403070511 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.568+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 msgr2=0x7f9c940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.568+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 0x7f9c940a4de0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f9ca4000c00 tx=0x7f9ca4005fb0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.568+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 msgr2=0x7f9ca81998f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.568+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f9c9800c8f0 tx=0x7f9c9800cc00 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.568+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 shutdown_connections 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f9c940a2920 0x7f9c940a4de0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9ca8105fc0 0x7f9ca81a4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9ca8100f70 0x7f9ca81993b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 --2- 172.21.15.73:0/69493889 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9ca8101920 0x7f9ca81998f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 >> 172.21.15.73:0/69493889 conn(0x7f9ca8076d30 msgr2=0x7f9ca8100490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 shutdown_connections 2024-03-09T00:05:42.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:42.569+0000 7f9cafe9d700 1 -- 172.21.15.73:0/69493889 wait complete. 2024-03-09T00:05:43.258 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:43 smithi102 ceph-mon[36106]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:43.259 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/69493889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-09T00:05:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:43 smithi179 ceph-mon[36637]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/69493889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-09T00:05:43.381 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:43 smithi073 ceph-mon[32309]: pgmap v340: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:43.381 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/69493889' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 -- 172.21.15.73:0/2615874578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e006a430 msgr2=0x7f24e010ec70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2615874578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e006a430 0x7f24e010ec70 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f24d800b0d0 tx=0x7f24d800b3e0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 -- 172.21.15.73:0/2615874578 shutdown_connections 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2615874578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e006a430 0x7f24e010ec70 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2615874578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e0107e30 0x7f24e0108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2615874578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0069a90 0x7f24e0069ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 -- 172.21.15.73:0/2615874578 >> 172.21.15.73:0/2615874578 conn(0x7f24e0077c10 msgr2=0x7f24e0078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 -- 172.21.15.73:0/2615874578 shutdown_connections 2024-03-09T00:05:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.909+0000 7f24e7c28700 1 -- 172.21.15.73:0/2615874578 wait complete. 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 Processor -- start 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 -- start start 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 0x7f24e0198970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 0x7f24e019cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0112080 con 0x7f24e0069a90 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0111f00 con 0x7f24e006a430 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e7c28700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0112200 con 0x7f24e0107e30 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e59c4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e59c4700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51304/0 (socket says 172.21.15.73:51304) 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e59c4700 1 -- 172.21.15.73:0/2238213376 learned_addr learned my addr 172.21.15.73:0/2238213376 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:44.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e51c3700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 0x7f24e0198970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e61c5700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 0x7f24e019cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e59c4700 1 -- 172.21.15.73:0/2238213376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 msgr2=0x7f24e019cd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.910+0000 7f24e59c4700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 0x7f24e019cd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e59c4700 1 -- 172.21.15.73:0/2238213376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 msgr2=0x7f24e0198970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e59c4700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 0x7f24e0198970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e59c4700 1 -- 172.21.15.73:0/2238213376 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e019d440 con 0x7f24e0069a90 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e61c5700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 0x7f24e019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e51c3700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 0x7f24e0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e59c4700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c9b0 tx=0x7f24d400cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4007b00 con 0x7f24e0069a90 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24d4007ca0 con 0x7f24e0069a90 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e019d730 con 0x7f24e0069a90 2024-03-09T00:05:44.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24d4018780 con 0x7f24e0069a90 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.911+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e01a4fb0 con 0x7f24e0069a90 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.912+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e0068aa0 con 0x7f24e0069a90 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.913+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f24d400f450 con 0x7f24e0069a90 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.913+0000 7f24d2ffd700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 0x7f24cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.914+0000 7f24e51c3700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 0x7f24cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.914+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f24d40cf1e0 con 0x7f24e0069a90 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.914+0000 7f24e51c3700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 0x7f24cc0a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f24dc005950 tx=0x7f24dc0058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:44.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:43.917+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f24d4014030 con 0x7f24e0069a90 2024-03-09T00:05:44.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.135+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f24e0199790 con 0x7f24e0069a90 2024-03-09T00:05:44.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.136+0000 7f24d2ffd700 1 -- 172.21.15.73:0/2238213376 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f24d409be50 con 0x7f24e0069a90 2024-03-09T00:05:44.131 INFO:teuthology.orchestra.run.smithi073.stdout:219043332148 2024-03-09T00:05:44.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 msgr2=0x7f24cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 0x7f24cc0a4de0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f24dc005950 tx=0x7f24dc0058e0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 msgr2=0x7f24e0198430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f24d400c9b0 tx=0x7f24d400cd70 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 shutdown_connections 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24e0069a90 0x7f24e0198430 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24cc0a2920 0x7f24cc0a4de0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24e0107e30 0x7f24e019cd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 --2- 172.21.15.73:0/2238213376 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24e006a430 0x7f24e0198970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 >> 172.21.15.73:0/2238213376 conn(0x7f24e0077c10 msgr2=0x7f24e00ff730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 shutdown_connections 2024-03-09T00:05:44.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.138+0000 7f24e7c28700 1 -- 172.21.15.73:0/2238213376 wait complete. 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- 172.21.15.73:0/2909078787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 msgr2=0x7fc4a010db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 --2- 172.21.15.73:0/2909078787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a010db60 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fc49800b0a0 tx=0x7fc49800b3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- 172.21.15.73:0/2909078787 shutdown_connections 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 --2- 172.21.15.73:0/2909078787 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a010db60 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 --2- 172.21.15.73:0/2909078787 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0106d20 0x7fc4a0107100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 --2- 172.21.15.73:0/2909078787 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0100ce0 0x7fc4a0101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- 172.21.15.73:0/2909078787 >> 172.21.15.73:0/2909078787 conn(0x7fc4a0077c10 msgr2=0x7fc4a0078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- 172.21.15.73:0/2909078787 shutdown_connections 2024-03-09T00:05:44.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- 172.21.15.73:0/2909078787 wait complete. 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 Processor -- start 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.263+0000 7fc4a6874700 1 -- start start 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0100ce0 0x7fc4a0198550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 0x7fc4a019ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0110fa0 con 0x7fc4a0101680 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0110e20 con 0x7fc4a0100ce0 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a6874700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0111120 con 0x7fc4a0106d20 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51312/0 (socket says 172.21.15.73:51312) 2024-03-09T00:05:44.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 -- 172.21.15.73:0/1354333750 learned_addr learned my addr 172.21.15.73:0/1354333750 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49ffff700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0100ce0 0x7fc4a0198550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc4a4e11700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 0x7fc4a019ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 -- 172.21.15.73:0/1354333750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 msgr2=0x7fc4a019ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 0x7fc4a019ce20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.264+0000 7fc49f7fe700 1 -- 172.21.15.73:0/1354333750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0100ce0 msgr2=0x7fc4a0198550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49f7fe700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0100ce0 0x7fc4a0198550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49f7fe700 1 -- 172.21.15.73:0/1354333750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4a019d560 con 0x7fc4a0101680 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc4a4e11700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 0x7fc4a019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49f7fe700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc494007aa0 con 0x7fc4a0101680 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a019d850 con 0x7fc4a0101680 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc494004d10 con 0x7fc4a0101680 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc49400f450 con 0x7fc4a0101680 2024-03-09T00:05:44.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.265+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a01a50d0 con 0x7fc4a0101680 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.266+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0068aa0 con 0x7fc4a0101680 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.267+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fc494004750 con 0x7fc4a0101680 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.267+0000 7fc49d7fa700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 0x7fc4880a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.268+0000 7fc49ffff700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 0x7fc4880a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.268+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fc4940ced50 con 0x7fc4a0101680 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.268+0000 7fc49ffff700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 0x7fc4880a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc490005950 tx=0x7fc49000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:44.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.272+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc49409b9c0 con 0x7fc4a0101680 2024-03-09T00:05:44.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.472+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fc4a01a5360 con 0x7fc4a0101680 2024-03-09T00:05:44.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.473+0000 7fc49d7fa700 1 -- 172.21.15.73:0/1354333750 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fc494005700 con 0x7fc4a0101680 2024-03-09T00:05:44.468 INFO:teuthology.orchestra.run.smithi073.stdout:34359738456 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 msgr2=0x7fc4880a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 0x7fc4880a4de0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7fc490005950 tx=0x7fc49000b410 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 msgr2=0x7fc4a0198a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fc49400c970 tx=0x7fc49400cc80 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 shutdown_connections 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc4880a2920 0x7fc4880a4de0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc4a0106d20 0x7fc4a019ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc4a0100ce0 0x7fc4a0198550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 --2- 172.21.15.73:0/1354333750 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc4a0101680 0x7fc4a0198a90 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 >> 172.21.15.73:0/1354333750 conn(0x7fc4a0077c10 msgr2=0x7fc4a00fffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 shutdown_connections 2024-03-09T00:05:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:44.475+0000 7fc4a6874700 1 -- 172.21.15.73:0/1354333750 wait complete. 2024-03-09T00:05:45.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:45 smithi073 ceph-mon[32309]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:45.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:45 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2238213376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-09T00:05:45.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:45 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1354333750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-09T00:05:45.305 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738451 got 34359738456 for osd.0 2024-03-09T00:05:45.306 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:45 smithi179 ceph-mon[36637]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:45 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2238213376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-09T00:05:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:45 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1354333750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-09T00:05:45.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:45 smithi102 ceph-mon[36106]: pgmap v341: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:45.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:45 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2238213376' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-09T00:05:45.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:45 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1354333750' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-09T00:05:46.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 -- 172.21.15.73:0/171829844 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810106d20 msgr2=0x7f8810107100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:46.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 --2- 172.21.15.73:0/171829844 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810106d20 0x7f8810107100 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f8804009990 tx=0x7f8804009ca0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 -- 172.21.15.73:0/171829844 shutdown_connections 2024-03-09T00:05:46.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 --2- 172.21.15.73:0/171829844 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810106d20 0x7f8810107100 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 --2- 172.21.15.73:0/171829844 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810101680 0x7f881010db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 --2- 172.21.15.73:0/171829844 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 0x7f8810101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 -- 172.21.15.73:0/171829844 >> 172.21.15.73:0/171829844 conn(0x7f8810077c10 msgr2=0x7f8810078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:46.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 -- 172.21.15.73:0/171829844 shutdown_connections 2024-03-09T00:05:46.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.347+0000 7f8816c69700 1 -- 172.21.15.73:0/171829844 wait complete. 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 Processor -- start 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 -- start start 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 0x7f88101984f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 0x7f881019cdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88101120c0 con 0x7f8810101680 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8810111f40 con 0x7f8810106d20 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8816c69700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8810112240 con 0x7f8810100ce0 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f880ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f880ffff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51342/0 (socket says 172.21.15.73:51342) 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f880ffff700 1 -- 172.21.15.73:0/1751215176 learned_addr learned my addr 172.21.15.73:0/1751215176 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:46.345 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8814a05700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 0x7f88101984f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.348+0000 7f8815206700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 0x7f881019cdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 -- 172.21.15.73:0/1751215176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 msgr2=0x7f88101984f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 0x7f88101984f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 -- 172.21.15.73:0/1751215176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 msgr2=0x7f881019cdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 0x7f881019cdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 -- 172.21.15.73:0/1751215176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f881019d500 con 0x7f8810101680 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f8815206700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 0x7f881019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880ffff700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00d980 tx=0x7f87fc00dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc017070 con 0x7f8810101680 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87fc005020 con 0x7f8810101680 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f881019d7f0 con 0x7f8810101680 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87fc009bc0 con 0x7f8810101680 2024-03-09T00:05:46.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.349+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88101a5070 con 0x7f8810101680 2024-03-09T00:05:46.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.350+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8810068aa0 con 0x7f8810101680 2024-03-09T00:05:46.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.351+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f87fc0051d0 con 0x7f8810101680 2024-03-09T00:05:46.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.351+0000 7f880dffb700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 0x7f88000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:46.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.352+0000 7f8814a05700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 0x7f88000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:46.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.352+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f87fc0cee40 con 0x7f8810101680 2024-03-09T00:05:46.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.352+0000 7f8814a05700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 0x7f88000a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f880400b5c0 tx=0x7f8804005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:46.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.356+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f87fc020020 con 0x7f8810101680 2024-03-09T00:05:46.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.558+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f881002d830 con 0x7f8810101680 2024-03-09T00:05:46.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.559+0000 7f880dffb700 1 -- 172.21.15.73:0/1751215176 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f87fc0b9e20 con 0x7f8810101680 2024-03-09T00:05:46.553 INFO:teuthology.orchestra.run.smithi073.stdout:309237645350 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 msgr2=0x7f88000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 0x7f88000a4e60 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f880400b5c0 tx=0x7f8804005960 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 msgr2=0x7f8810198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f87fc00d980 tx=0x7f87fc00dc90 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 shutdown_connections 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f88000a29a0 0x7f88000a4e60 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8810106d20 0x7f881019cdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8810100ce0 0x7f88101984f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 --2- 172.21.15.73:0/1751215176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8810101680 0x7f8810198a30 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 >> 172.21.15.73:0/1751215176 conn(0x7f8810077c10 msgr2=0x7f88100ff970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:46.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 shutdown_connections 2024-03-09T00:05:46.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:46.561+0000 7f8816c69700 1 -- 172.21.15.73:0/1751215176 wait complete. 2024-03-09T00:05:46.850 INFO:tasks.cephadm.ceph_manager.ceph:need seq 219043332142 got 219043332148 for osd.7 2024-03-09T00:05:46.850 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:47.077 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:47 smithi073 ceph-mon[32309]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:47.077 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:47 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1751215176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-09T00:05:47.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 -- 172.21.15.73:0/1927149522 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00100f70 msgr2=0x7f7a00101350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 --2- 172.21.15.73:0/1927149522 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00100f70 0x7f7a00101350 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f79fc0099c0 tx=0x7f79fc009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 -- 172.21.15.73:0/1927149522 shutdown_connections 2024-03-09T00:05:47.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 --2- 172.21.15.73:0/1927149522 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00100f70 0x7f7a00101350 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 --2- 172.21.15.73:0/1927149522 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00105fc0 0x7f7a0010db90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 --2- 172.21.15.73:0/1927149522 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00101920 0x7f7a001058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.263+0000 7f7a073fa700 1 -- 172.21.15.73:0/1927149522 >> 172.21.15.73:0/1927149522 conn(0x7f7a00076d30 msgr2=0x7f7a00077140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- 172.21.15.73:0/1927149522 shutdown_connections 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- 172.21.15.73:0/1927149522 wait complete. 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 Processor -- start 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- start start 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 0x7f7a001993d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00101920 0x7f7a00199910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a00110fb0 con 0x7f7a00105fc0 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a00110e30 con 0x7f7a00100f70 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a073fa700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a00111130 con 0x7f7a00101920 2024-03-09T00:05:47.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.264+0000 7f7a05997700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51358/0 (socket says 172.21.15.73:51358) 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 -- 172.21.15.73:0/332289688 learned_addr learned my addr 172.21.15.73:0/332289688 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a04995700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00101920 0x7f7a00199910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05196700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 0x7f7a001993d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 -- 172.21.15.73:0/332289688 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00101920 msgr2=0x7f7a00199910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00101920 0x7f7a00199910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 -- 172.21.15.73:0/332289688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 msgr2=0x7f7a001993d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 0x7f7a001993d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 -- 172.21.15.73:0/332289688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a001a4990 con 0x7f7a00105fc0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05196700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 0x7f7a001993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a05997700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f79f8007ee0 tx=0x7f79f800d3b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f800dbd0 con 0x7f7a00105fc0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f79f800dd70 con 0x7f7a00105fc0 2024-03-09T00:05:47.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a001a4bc0 con 0x7f7a00105fc0 2024-03-09T00:05:47.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f79f8005a60 con 0x7f7a00105fc0 2024-03-09T00:05:47.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.265+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a001a50b0 con 0x7f7a00105fc0 2024-03-09T00:05:47.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.266+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a00068aa0 con 0x7f7a00105fc0 2024-03-09T00:05:47.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.267+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f79f801a020 con 0x7f7a00105fc0 2024-03-09T00:05:47.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.268+0000 7f79f67fc700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.268+0000 7f7a05196700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.268+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f79f80ceca0 con 0x7f7a00105fc0 2024-03-09T00:05:47.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.268+0000 7f7a05196700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 0x7f79ec0a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009750 tx=0x7f79fc009f10 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:47.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.271+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f79f8010070 con 0x7f7a00105fc0 2024-03-09T00:05:47.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:47 smithi179 ceph-mon[36637]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:47.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:47 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1751215176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-09T00:05:47.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 -- 172.21.15.73:0/1658234613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc70069a90 msgr2=0x7fdc70069ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1658234613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc70069a90 0x7fdc70069ef0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7fdc600099c0 tx=0x7fdc60009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 -- 172.21.15.73:0/1658234613 shutdown_connections 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1658234613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc7006a430 0x7fdc7010ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1658234613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc70108210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1658234613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc70069a90 0x7fdc70069ef0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.457+0000 7fdc776d3700 1 -- 172.21.15.73:0/1658234613 >> 172.21.15.73:0/1658234613 conn(0x7fdc70077c10 msgr2=0x7fdc70078020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- 172.21.15.73:0/1658234613 shutdown_connections 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- 172.21.15.73:0/1658234613 wait complete. 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 Processor -- start 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- start start 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 0x7fdc70198540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc7019ce10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc701120b0 con 0x7fdc7006a430 2024-03-09T00:05:47.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70111f30 con 0x7fdc70069a90 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc776d3700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc70112230 con 0x7fdc70107e30 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.458+0000 7fdc74c6e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51380/0 (socket says 172.21.15.73:51380) 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc75c70700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc7019ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 -- 172.21.15.73:0/1640688832 learned_addr learned my addr 172.21.15.73:0/1640688832 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:47.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc7546f700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 0x7fdc70198540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 -- 172.21.15.73:0/1640688832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 msgr2=0x7fdc7019ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc7019ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 -- 172.21.15.73:0/1640688832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 msgr2=0x7fdc70198540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 0x7fdc70198540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.456 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 -- 172.21.15.73:0/1640688832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc7019d550 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc7546f700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 0x7fdc70198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc75c70700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc7019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc74c6e700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc60006010 tx=0x7fdc60009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60026070 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc7019d7e0 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc60004cd0 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc60003ba0 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.459+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc701a50c0 con 0x7fdc7006a430 2024-03-09T00:05:47.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.460+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc70068aa0 con 0x7fdc7006a430 2024-03-09T00:05:47.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.462+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fdc60029070 con 0x7fdc7006a430 2024-03-09T00:05:47.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.463+0000 7fdc667fc700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 0x7fdc5c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:47.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.463+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fdc600cfd90 con 0x7fdc7006a430 2024-03-09T00:05:47.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.463+0000 7fdc7546f700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 0x7fdc5c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:47.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.464+0000 7fdc7546f700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 0x7fdc5c0a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0103d0 tx=0x7fdc6c00b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:47.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.466+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdc60013070 con 0x7fdc7006a430 2024-03-09T00:05:47.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.483+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f7a0002d9f0 con 0x7f7a00105fc0 2024-03-09T00:05:47.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.484+0000 7f79f67fc700 1 -- 172.21.15.73:0/332289688 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f79f809b880 con 0x7f7a00105fc0 2024-03-09T00:05:47.479 INFO:teuthology.orchestra.run.smithi073.stdout:287762808875 2024-03-09T00:05:47.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 msgr2=0x7f79ec0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 0x7f79ec0a4de0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f79fc009750 tx=0x7f79fc009f10 comp rx=0 tx=0).stop 2024-03-09T00:05:47.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 msgr2=0x7f7a001a4450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f79f8007ee0 tx=0x7f79f800d3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 shutdown_connections 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a00105fc0 0x7f7a001a4450 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f79ec0a2920 0x7f79ec0a4de0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a00100f70 0x7f7a001993d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.487+0000 7f7a073fa700 1 --2- 172.21.15.73:0/332289688 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a00101920 0x7f7a00199910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.488+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 >> 172.21.15.73:0/332289688 conn(0x7f7a00076d30 msgr2=0x7f7a001004b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.488+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 shutdown_connections 2024-03-09T00:05:47.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.488+0000 7f7a073fa700 1 -- 172.21.15.73:0/332289688 wait complete. 2024-03-09T00:05:47.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:47 smithi102 ceph-mon[36106]: pgmap v342: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:47.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:47 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1751215176' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-09T00:05:47.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.680+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7fdc7002d830 con 0x7fdc7006a430 2024-03-09T00:05:47.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.681+0000 7fdc667fc700 1 -- 172.21.15.73:0/1640688832 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fdc6009ca00 con 0x7fdc7006a430 2024-03-09T00:05:47.676 INFO:teuthology.orchestra.run.smithi073.stdout:55834574931 2024-03-09T00:05:47.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 msgr2=0x7fdc5c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 0x7fdc5c0a4de0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fdc6c0103d0 tx=0x7fdc6c00b3c0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 msgr2=0x7fdc70198a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7fdc60006010 tx=0x7fdc60009f80 comp rx=0 tx=0).stop 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 shutdown_connections 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fdc5c0a2920 0x7fdc5c0a4de0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdc70107e30 0x7fdc7019ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdc70069a90 0x7fdc70198540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 --2- 172.21.15.73:0/1640688832 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdc7006a430 0x7fdc70198a80 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 >> 172.21.15.73:0/1640688832 conn(0x7fdc70077c10 msgr2=0x7fdc700ff850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 shutdown_connections 2024-03-09T00:05:47.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:47.683+0000 7fdc776d3700 1 -- 172.21.15.73:0/1640688832 wait complete. 2024-03-09T00:05:48.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:48 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/332289688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-09T00:05:48.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:48 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:48 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1640688832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-09T00:05:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:48 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/332289688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-09T00:05:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:48 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:48 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1640688832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-09T00:05:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:48 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/332289688' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-09T00:05:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:48 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:48 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1640688832' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-09T00:05:48.590 INFO:tasks.cephadm.ceph_manager.ceph:need seq 309237645343 got 309237645350 for osd.10 2024-03-09T00:05:48.590 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:49.278 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:49 smithi073 ceph-mon[32309]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:49.278 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:49 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:05:49.278 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:49 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:05:49.278 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:49 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:49.304 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574925 got 55834574931 for osd.1 2024-03-09T00:05:49.304 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:49.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:49 smithi179 ceph-mon[36637]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:49.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:49 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:05:49.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:49 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:05:49.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:49 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:49.469 INFO:tasks.cephadm.ceph_manager.ceph:need seq 253403070504 got 253403070511 for osd.8 2024-03-09T00:05:49.470 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:49.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:49 smithi102 ceph-mon[36106]: pgmap v343: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:49.545 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:49 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:05:49.545 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:49 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:05:49.545 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:49 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' 2024-03-09T00:05:49.757 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:50.194 INFO:tasks.cephadm.ceph_manager.ceph:need seq 287762808870 got 287762808875 for osd.9 2024-03-09T00:05:50.194 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:51.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:51 smithi179 ceph-mon[36637]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:51.343 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:51 smithi073 ceph-mon[32309]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:51.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:51 smithi102 ceph-mon[36106]: pgmap v344: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 -- 172.21.15.73:0/949605072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 msgr2=0x7f509406a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 --2- 172.21.15.73:0/949605072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f509406a940 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f50880099c0 tx=0x7f5088009cd0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 -- 172.21.15.73:0/949605072 shutdown_connections 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 --2- 172.21.15.73:0/949605072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 0x7f509410eca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 --2- 172.21.15.73:0/949605072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 0x7f5094108240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 --2- 172.21.15.73:0/949605072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f509406a940 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 -- 172.21.15.73:0/949605072 >> 172.21.15.73:0/949605072 conn(0x7f5094076d00 msgr2=0x7f5094077110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 -- 172.21.15.73:0/949605072 shutdown_connections 2024-03-09T00:05:51.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.910+0000 7f5098e72700 1 -- 172.21.15.73:0/949605072 wait complete. 2024-03-09T00:05:51.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 Processor -- start 2024-03-09T00:05:51.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 -- start start 2024-03-09T00:05:51.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:51.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 0x7f5094101320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:51.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 0x7f5094101860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50941120c0 con 0x7f509406a4e0 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094111f40 con 0x7f5094107e60 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5098e72700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5094112240 con 0x7f509406ae80 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43890/0 (socket says 172.21.15.73:43890) 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 -- 172.21.15.73:0/3358081473 learned_addr learned my addr 172.21.15.73:0/3358081473 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5091d9b700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 0x7f5094101320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 -- 172.21.15.73:0/3358081473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 msgr2=0x7f5094101320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 0x7f5094101320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f5092d9d700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 0x7f5094101860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 -- 172.21.15.73:0/3358081473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 msgr2=0x7f5094101860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 0x7f5094101860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 -- 172.21.15.73:0/3358081473 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50941a6b20 con 0x7f509406a4e0 2024-03-09T00:05:51.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.911+0000 7f509259c700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f508400c920 tx=0x7f508400cce0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.912+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5084007b00 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.912+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5084004d10 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.912+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50941a6e10 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.912+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f508400f450 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.912+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50941a72c0 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.913+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5094103510 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.914+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f5084004750 con 0x7f509406a4e0 2024-03-09T00:05:51.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.915+0000 7f50837fe700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 0x7f507c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:51.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.915+0000 7f5091d9b700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 0x7f507c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:51.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.915+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f50840ceef0 con 0x7f509406a4e0 2024-03-09T00:05:51.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.916+0000 7f5091d9b700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 0x7f507c0a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5088006010 tx=0x7f5088000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:51.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:51.919+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f50840fd260 con 0x7f509406a4e0 2024-03-09T00:05:52.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.116+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f509402d830 con 0x7f509406a4e0 2024-03-09T00:05:52.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.116+0000 7f50837fe700 1 -- 172.21.15.73:0/3358081473 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f5084014030 con 0x7f509406a4e0 2024-03-09T00:05:52.111 INFO:teuthology.orchestra.run.smithi073.stdout:343597383714 2024-03-09T00:05:52.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 msgr2=0x7f507c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:52.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 0x7f507c0a4de0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f5088006010 tx=0x7f5088000bc0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 msgr2=0x7f5094100de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f508400c920 tx=0x7f508400cce0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 shutdown_connections 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f509406a4e0 0x7f5094100de0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f507c0a2920 0x7f507c0a4de0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5094107e60 0x7f5094101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 --2- 172.21.15.73:0/3358081473 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f509406ae80 0x7f5094101320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 >> 172.21.15.73:0/3358081473 conn(0x7f5094076d00 msgr2=0x7f50940ffab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 shutdown_connections 2024-03-09T00:05:52.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:52.119+0000 7f5098e72700 1 -- 172.21.15.73:0/3358081473 wait complete. 2024-03-09T00:05:52.875 INFO:tasks.cephadm.ceph_manager.ceph:need seq 343597383708 got 343597383714 for osd.11 2024-03-09T00:05:52.875 DEBUG:teuthology.parallel:result is None 2024-03-09T00:05:52.875 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-09T00:05:52.876 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-09T00:05:53.123 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:53.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:53 smithi073 ceph-mon[32309]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:53.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:53 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3358081473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-09T00:05:53.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:53 smithi179 ceph-mon[36637]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:53.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:53 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3358081473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-09T00:05:53.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:53 smithi102 ceph-mon[36106]: pgmap v345: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:53.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:53 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3358081473' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- 172.21.15.73:0/1031878594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 msgr2=0x7f24a410db20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 --2- 172.21.15.73:0/1031878594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a410db20 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f249c00b0a0 tx=0x7f249c00b3b0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- 172.21.15.73:0/1031878594 shutdown_connections 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 --2- 172.21.15.73:0/1031878594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a410db20 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 --2- 172.21.15.73:0/1031878594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 0x7f24a4107150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 --2- 172.21.15.73:0/1031878594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 0x7f24a4101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- 172.21.15.73:0/1031878594 >> 172.21.15.73:0/1031878594 conn(0x7f24a4077cf0 msgr2=0x7f24a4078100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:55.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- 172.21.15.73:0/1031878594 shutdown_connections 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- 172.21.15.73:0/1031878594 wait complete. 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 Processor -- start 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.094+0000 7f24abc17700 1 -- start start 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 0x7f24a419a670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 0x7f24a419ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4110fa0 con 0x7f24a4101680 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4110e20 con 0x7f24a4106d70 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4111120 con 0x7f24a4100cc0 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43916/0 (socket says 172.21.15.73:43916) 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 -- 172.21.15.73:0/3030941361 learned_addr learned my addr 172.21.15.73:0/3030941361 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24aa1b4700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 0x7f24a419ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:55.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a99b3700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 0x7f24a419a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 -- 172.21.15.73:0/3030941361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 msgr2=0x7f24a419a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 0x7f24a419a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 -- 172.21.15.73:0/3030941361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 msgr2=0x7f24a419ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 0x7f24a419ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 -- 172.21.15.73:0/3030941361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a419f680 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24a91b2700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f249400d980 tx=0x7f249400dc90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494017070 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2494005020 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a419f970 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2494009bc0 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.095+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a41a71f0 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.096+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a4068aa0 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.097+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f24940051d0 con 0x7f24a4101680 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.098+0000 7f249affd700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 0x7f24900a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:55.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.098+0000 7f24a99b3700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 0x7f24900a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:55.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.098+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f24940cee90 con 0x7f24a4101680 2024-03-09T00:05:55.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.098+0000 7f24a99b3700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 0x7f24900a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24a0005fd0 tx=0x7f24a000a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:55.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.103+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2494020020 con 0x7f24a4101680 2024-03-09T00:05:55.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.296+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f24a406ac10 con 0x7f24900a2920 2024-03-09T00:05:55.291 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:55 smithi073 ceph-mon[32309]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:55.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.306+0000 7f249affd700 1 -- 172.21.15.73:0/3030941361 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340270 (secure 0 0 0) 0x7f24a406ac10 con 0x7f24900a2920 2024-03-09T00:05:55.306 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 msgr2=0x7f24900a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 0x7f24900a4de0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f24a0005fd0 tx=0x7f24a000a560 comp rx=0 tx=0).stop 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 msgr2=0x7f24a419abb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f249400d980 tx=0x7f249400dc90 comp rx=0 tx=0).stop 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 shutdown_connections 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24900a2920 0x7f24900a4de0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24a4106d70 0x7f24a419ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24a4100cc0 0x7f24a419a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 --2- 172.21.15.73:0/3030941361 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24a4101680 0x7f24a419abb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.309+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 >> 172.21.15.73:0/3030941361 conn(0x7f24a4077cf0 msgr2=0x7f24a406a500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.310+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 shutdown_connections 2024-03-09T00:05:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:55.310+0000 7f24abc17700 1 -- 172.21.15.73:0/3030941361 wait complete. 2024-03-09T00:05:55.310 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:05:55.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:55 smithi179 ceph-mon[36637]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:55.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:55 smithi102 ceph-mon[36106]: pgmap v346: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:56.147 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":346,"stamp":"2024-03-09T00:05:53.955002+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":291,"num_read_kb":261,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388140,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553684,"statfs":{"total":1151940427776,"available":1151542972416,"internally_reserved":0,"allocated":14286848,"data_stored":7950170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002944"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055886+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:05:05.055886+0000","last_peered":"2024-03-09T00:05:05.055886+0000","last_clean":"2024-03-09T00:05:05.055886+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:05:05.055886+0000","last_undegraded":"2024-03-09T00:05:05.055886+0000","last_fullsized":"2024-03-09T00:05:05.055886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136779+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:05:05.136779+0000","last_peered":"2024-03-09T00:05:05.136779+0000","last_clean":"2024-03-09T00:05:05.136779+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:05:05.136779+0000","last_undegraded":"2024-03-09T00:05:05.136779+0000","last_fullsized":"2024-03-09T00:05:05.136779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136593+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:05:05.136593+0000","last_peered":"2024-03-09T00:05:05.136593+0000","last_clean":"2024-03-09T00:05:05.136593+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:05:05.136593+0000","last_undegraded":"2024-03-09T00:05:05.136593+0000","last_fullsized":"2024-03-09T00:05:05.136593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138656+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:05:05.138656+0000","last_peered":"2024-03-09T00:05:05.138656+0000","last_clean":"2024-03-09T00:05:05.138656+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:05:05.138656+0000","last_undegraded":"2024-03-09T00:05:05.138656+0000","last_fullsized":"2024-03-09T00:05:05.138656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050159+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:05:05.050159+0000","last_peered":"2024-03-09T00:05:05.050159+0000","last_clean":"2024-03-09T00:05:05.050159+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:05:05.050159+0000","last_undegraded":"2024-03-09T00:05:05.050159+0000","last_fullsized":"2024-03-09T00:05:05.050159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058033+0000","last_change":"2024-03-09T00:05:05.058033+0000","last_active":"2024-03-09T00:05:05.058033+0000","last_peered":"2024-03-09T00:05:05.058033+0000","last_clean":"2024-03-09T00:05:05.058033+0000","last_became_active":"2024-03-09T00:05:05.057575+0000","last_became_peered":"2024-03-09T00:05:05.057575+0000","last_unstale":"2024-03-09T00:05:05.058033+0000","last_undegraded":"2024-03-09T00:05:05.058033+0000","last_fullsized":"2024-03-09T00:05:05.058033+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:35:58.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056728+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:05:05.056728+0000","last_peered":"2024-03-09T00:05:05.056728+0000","last_clean":"2024-03-09T00:05:05.056728+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:05:05.056728+0000","last_undegraded":"2024-03-09T00:05:05.056728+0000","last_fullsized":"2024-03-09T00:05:05.056728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049165+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:05:05.049165+0000","last_peered":"2024-03-09T00:05:05.049165+0000","last_clean":"2024-03-09T00:05:05.049165+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:05:05.049165+0000","last_undegraded":"2024-03-09T00:05:05.049165+0000","last_fullsized":"2024-03-09T00:05:05.049165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056946+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:05:05.056946+0000","last_peered":"2024-03-09T00:05:05.056946+0000","last_clean":"2024-03-09T00:05:05.056946+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:05:05.056946+0000","last_undegraded":"2024-03-09T00:05:05.056946+0000","last_fullsized":"2024-03-09T00:05:05.056946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054029+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:05:05.054029+0000","last_peered":"2024-03-09T00:05:05.054029+0000","last_clean":"2024-03-09T00:05:05.054029+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:05:05.054029+0000","last_undegraded":"2024-03-09T00:05:05.054029+0000","last_fullsized":"2024-03-09T00:05:05.054029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056580+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:05:05.056580+0000","last_peered":"2024-03-09T00:05:05.056580+0000","last_clean":"2024-03-09T00:05:05.056580+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:05:05.056580+0000","last_undegraded":"2024-03-09T00:05:05.056580+0000","last_fullsized":"2024-03-09T00:05:05.056580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138418+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:05:05.138418+0000","last_peered":"2024-03-09T00:05:05.138418+0000","last_clean":"2024-03-09T00:05:05.138418+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:05:05.138418+0000","last_undegraded":"2024-03-09T00:05:05.138418+0000","last_fullsized":"2024-03-09T00:05:05.138418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052287+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:05:05.052287+0000","last_peered":"2024-03-09T00:05:05.052287+0000","last_clean":"2024-03-09T00:05:05.052287+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:05:05.052287+0000","last_undegraded":"2024-03-09T00:05:05.052287+0000","last_fullsized":"2024-03-09T00:05:05.052287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136435+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:05:05.136435+0000","last_peered":"2024-03-09T00:05:05.136435+0000","last_clean":"2024-03-09T00:05:05.136435+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:05:05.136435+0000","last_undegraded":"2024-03-09T00:05:05.136435+0000","last_fullsized":"2024-03-09T00:05:05.136435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136587+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:05:05.136587+0000","last_peered":"2024-03-09T00:05:05.136587+0000","last_clean":"2024-03-09T00:05:05.136587+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:05:05.136587+0000","last_undegraded":"2024-03-09T00:05:05.136587+0000","last_fullsized":"2024-03-09T00:05:05.136587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055637+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:05:05.055637+0000","last_peered":"2024-03-09T00:05:05.055637+0000","last_clean":"2024-03-09T00:05:05.055637+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:05:05.055637+0000","last_undegraded":"2024-03-09T00:05:05.055637+0000","last_fullsized":"2024-03-09T00:05:05.055637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055631+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:05:05.055631+0000","last_peered":"2024-03-09T00:05:05.055631+0000","last_clean":"2024-03-09T00:05:05.055631+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:05:05.055631+0000","last_undegraded":"2024-03-09T00:05:05.055631+0000","last_fullsized":"2024-03-09T00:05:05.055631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051647+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:05:05.051647+0000","last_peered":"2024-03-09T00:05:05.051647+0000","last_clean":"2024-03-09T00:05:05.051647+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:05:05.051647+0000","last_undegraded":"2024-03-09T00:05:05.051647+0000","last_fullsized":"2024-03-09T00:05:05.051647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053008+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:05:05.053008+0000","last_peered":"2024-03-09T00:05:05.053008+0000","last_clean":"2024-03-09T00:05:05.053008+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:05:05.053008+0000","last_undegraded":"2024-03-09T00:05:05.053008+0000","last_fullsized":"2024-03-09T00:05:05.053008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138220+0000","last_change":"2024-03-09T00:05:05.138220+0000","last_active":"2024-03-09T00:05:05.138220+0000","last_peered":"2024-03-09T00:05:05.138220+0000","last_clean":"2024-03-09T00:05:05.138220+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:05:05.138220+0000","last_undegraded":"2024-03-09T00:05:05.138220+0000","last_fullsized":"2024-03-09T00:05:05.138220+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:41.455601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051852+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:05:05.051852+0000","last_peered":"2024-03-09T00:05:05.051852+0000","last_clean":"2024-03-09T00:05:05.051852+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:05:05.051852+0000","last_undegraded":"2024-03-09T00:05:05.051852+0000","last_fullsized":"2024-03-09T00:05:05.051852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049674+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:05:05.049674+0000","last_peered":"2024-03-09T00:05:05.049674+0000","last_clean":"2024-03-09T00:05:05.049674+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:05:05.049674+0000","last_undegraded":"2024-03-09T00:05:05.049674+0000","last_fullsized":"2024-03-09T00:05:05.049674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.137818+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:05:05.137818+0000","last_peered":"2024-03-09T00:05:05.137818+0000","last_clean":"2024-03-09T00:05:05.137818+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:05:05.137818+0000","last_undegraded":"2024-03-09T00:05:05.137818+0000","last_fullsized":"2024-03-09T00:05:05.137818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049925+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:05:05.049925+0000","last_peered":"2024-03-09T00:05:05.049925+0000","last_clean":"2024-03-09T00:05:05.049925+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:05:05.049925+0000","last_undegraded":"2024-03-09T00:05:05.049925+0000","last_fullsized":"2024-03-09T00:05:05.049925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136845+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:05:05.136845+0000","last_peered":"2024-03-09T00:05:05.136845+0000","last_clean":"2024-03-09T00:05:05.136845+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:05:05.136845+0000","last_undegraded":"2024-03-09T00:05:05.136845+0000","last_fullsized":"2024-03-09T00:05:05.136845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138164+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:05:05.138164+0000","last_peered":"2024-03-09T00:05:05.138164+0000","last_clean":"2024-03-09T00:05:05.138164+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:05:05.138164+0000","last_undegraded":"2024-03-09T00:05:05.138164+0000","last_fullsized":"2024-03-09T00:05:05.138164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.142560+0000","last_change":"2024-03-09T00:05:05.142560+0000","last_active":"2024-03-09T00:05:05.142560+0000","last_peered":"2024-03-09T00:05:05.142560+0000","last_clean":"2024-03-09T00:05:05.142560+0000","last_became_active":"2024-03-09T00:05:05.139877+0000","last_became_peered":"2024-03-09T00:05:05.139877+0000","last_unstale":"2024-03-09T00:05:05.142560+0000","last_undegraded":"2024-03-09T00:05:05.142560+0000","last_fullsized":"2024-03-09T00:05:05.142560+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:30.441723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056923+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:05:05.056923+0000","last_peered":"2024-03-09T00:05:05.056923+0000","last_clean":"2024-03-09T00:05:05.056923+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:05:05.056923+0000","last_undegraded":"2024-03-09T00:05:05.056923+0000","last_fullsized":"2024-03-09T00:05:05.056923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138277+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:05:05.138277+0000","last_peered":"2024-03-09T00:05:05.138277+0000","last_clean":"2024-03-09T00:05:05.138277+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:05:05.138277+0000","last_undegraded":"2024-03-09T00:05:05.138277+0000","last_fullsized":"2024-03-09T00:05:05.138277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050427+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:05:05.050427+0000","last_peered":"2024-03-09T00:05:05.050427+0000","last_clean":"2024-03-09T00:05:05.050427+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:05:05.050427+0000","last_undegraded":"2024-03-09T00:05:05.050427+0000","last_fullsized":"2024-03-09T00:05:05.050427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054482+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:05:05.054482+0000","last_peered":"2024-03-09T00:05:05.054482+0000","last_clean":"2024-03-09T00:05:05.054482+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:05:05.054482+0000","last_undegraded":"2024-03-09T00:05:05.054482+0000","last_fullsized":"2024-03-09T00:05:05.054482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052692+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:05:05.052692+0000","last_peered":"2024-03-09T00:05:05.052692+0000","last_clean":"2024-03-09T00:05:05.052692+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:05:05.052692+0000","last_undegraded":"2024-03-09T00:05:05.052692+0000","last_fullsized":"2024-03-09T00:05:05.052692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054835+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:05:05.054835+0000","last_peered":"2024-03-09T00:05:05.054835+0000","last_clean":"2024-03-09T00:05:05.054835+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:05:05.054835+0000","last_undegraded":"2024-03-09T00:05:05.054835+0000","last_fullsized":"2024-03-09T00:05:05.054835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049785+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:05:05.049785+0000","last_peered":"2024-03-09T00:05:05.049785+0000","last_clean":"2024-03-09T00:05:05.049785+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:05:05.049785+0000","last_undegraded":"2024-03-09T00:05:05.049785+0000","last_fullsized":"2024-03-09T00:05:05.049785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052486+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:05:05.052486+0000","last_peered":"2024-03-09T00:05:05.052486+0000","last_clean":"2024-03-09T00:05:05.052486+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:05:05.052486+0000","last_undegraded":"2024-03-09T00:05:05.052486+0000","last_fullsized":"2024-03-09T00:05:05.052486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053680+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:05:05.053680+0000","last_peered":"2024-03-09T00:05:05.053680+0000","last_clean":"2024-03-09T00:05:05.053680+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:05:05.053680+0000","last_undegraded":"2024-03-09T00:05:05.053680+0000","last_fullsized":"2024-03-09T00:05:05.053680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056415+0000","last_change":"2024-03-09T00:05:05.056415+0000","last_active":"2024-03-09T00:05:05.056415+0000","last_peered":"2024-03-09T00:05:05.056415+0000","last_clean":"2024-03-09T00:05:05.056415+0000","last_became_active":"2024-03-09T00:05:05.056025+0000","last_became_peered":"2024-03-09T00:05:05.056025+0000","last_unstale":"2024-03-09T00:05:05.056415+0000","last_undegraded":"2024-03-09T00:05:05.056415+0000","last_fullsized":"2024-03-09T00:05:05.056415+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:29.970623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055504+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:05:05.055504+0000","last_peered":"2024-03-09T00:05:05.055504+0000","last_clean":"2024-03-09T00:05:05.055504+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:05:05.055504+0000","last_undegraded":"2024-03-09T00:05:05.055504+0000","last_fullsized":"2024-03-09T00:05:05.055504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052237+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:05:05.052237+0000","last_peered":"2024-03-09T00:05:05.052237+0000","last_clean":"2024-03-09T00:05:05.052237+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:05:05.052237+0000","last_undegraded":"2024-03-09T00:05:05.052237+0000","last_fullsized":"2024-03-09T00:05:05.052237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049824+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:05:05.049824+0000","last_peered":"2024-03-09T00:05:05.049824+0000","last_clean":"2024-03-09T00:05:05.049824+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:05:05.049824+0000","last_undegraded":"2024-03-09T00:05:05.049824+0000","last_fullsized":"2024-03-09T00:05:05.049824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136618+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:05:05.136618+0000","last_peered":"2024-03-09T00:05:05.136618+0000","last_clean":"2024-03-09T00:05:05.136618+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:05:05.136618+0000","last_undegraded":"2024-03-09T00:05:05.136618+0000","last_fullsized":"2024-03-09T00:05:05.136618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053698+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:05:05.053698+0000","last_peered":"2024-03-09T00:05:05.053698+0000","last_clean":"2024-03-09T00:05:05.053698+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:05:05.053698+0000","last_undegraded":"2024-03-09T00:05:05.053698+0000","last_fullsized":"2024-03-09T00:05:05.053698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050966+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:05:05.050966+0000","last_peered":"2024-03-09T00:05:05.050966+0000","last_clean":"2024-03-09T00:05:05.050966+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:05:05.050966+0000","last_undegraded":"2024-03-09T00:05:05.050966+0000","last_fullsized":"2024-03-09T00:05:05.050966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049265+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:05:05.049265+0000","last_peered":"2024-03-09T00:05:05.049265+0000","last_clean":"2024-03-09T00:05:05.049265+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:05:05.049265+0000","last_undegraded":"2024-03-09T00:05:05.049265+0000","last_fullsized":"2024-03-09T00:05:05.049265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053532+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:05:05.053532+0000","last_peered":"2024-03-09T00:05:05.053532+0000","last_clean":"2024-03-09T00:05:05.053532+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:05:05.053532+0000","last_undegraded":"2024-03-09T00:05:05.053532+0000","last_fullsized":"2024-03-09T00:05:05.053532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056957+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:05:05.056957+0000","last_peered":"2024-03-09T00:05:05.056957+0000","last_clean":"2024-03-09T00:05:05.056957+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:05:05.056957+0000","last_undegraded":"2024-03-09T00:05:05.056957+0000","last_fullsized":"2024-03-09T00:05:05.056957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.140115+0000","last_change":"2024-03-09T00:05:05.139426+0000","last_active":"2024-03-09T00:05:05.140115+0000","last_peered":"2024-03-09T00:05:05.140115+0000","last_clean":"2024-03-09T00:05:05.140115+0000","last_became_active":"2024-03-09T00:05:05.138971+0000","last_became_peered":"2024-03-09T00:05:05.138971+0000","last_unstale":"2024-03-09T00:05:05.140115+0000","last_undegraded":"2024-03-09T00:05:05.140115+0000","last_fullsized":"2024-03-09T00:05:05.140115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:40:59.870972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057274+0000","last_change":"2024-03-09T00:04:13.334152+0000","last_active":"2024-03-09T00:05:05.057274+0000","last_peered":"2024-03-09T00:05:05.057274+0000","last_clean":"2024-03-09T00:05:05.057274+0000","last_became_active":"2024-03-09T00:04:13.333835+0000","last_became_peered":"2024-03-09T00:04:13.333835+0000","last_unstale":"2024-03-09T00:05:05.057274+0000","last_undegraded":"2024-03-09T00:05:05.057274+0000","last_fullsized":"2024-03-09T00:05:05.057274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:19:49.203289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136870+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:05:05.136870+0000","last_peered":"2024-03-09T00:05:05.136870+0000","last_clean":"2024-03-09T00:05:05.136870+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:05:05.136870+0000","last_undegraded":"2024-03-09T00:05:05.136870+0000","last_fullsized":"2024-03-09T00:05:05.136870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138319+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:05:05.138319+0000","last_peered":"2024-03-09T00:05:05.138319+0000","last_clean":"2024-03-09T00:05:05.138319+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:05:05.138319+0000","last_undegraded":"2024-03-09T00:05:05.138319+0000","last_fullsized":"2024-03-09T00:05:05.138319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054365+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:05:05.054365+0000","last_peered":"2024-03-09T00:05:05.054365+0000","last_clean":"2024-03-09T00:05:05.054365+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:05:05.054365+0000","last_undegraded":"2024-03-09T00:05:05.054365+0000","last_fullsized":"2024-03-09T00:05:05.054365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057072+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:05:05.057072+0000","last_peered":"2024-03-09T00:05:05.057072+0000","last_clean":"2024-03-09T00:05:05.057072+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:05:05.057072+0000","last_undegraded":"2024-03-09T00:05:05.057072+0000","last_fullsized":"2024-03-09T00:05:05.057072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058517+0000","last_change":"2024-03-09T00:05:05.058517+0000","last_active":"2024-03-09T00:05:05.058517+0000","last_peered":"2024-03-09T00:05:05.058517+0000","last_clean":"2024-03-09T00:05:05.058517+0000","last_became_active":"2024-03-09T00:05:05.057877+0000","last_became_peered":"2024-03-09T00:05:05.057877+0000","last_unstale":"2024-03-09T00:05:05.058517+0000","last_undegraded":"2024-03-09T00:05:05.058517+0000","last_fullsized":"2024-03-09T00:05:05.058517+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:42:49.424790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'30","reported_seq":206,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:49.109306+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:05:49.109306+0000","last_peered":"2024-03-09T00:05:49.109306+0000","last_clean":"2024-03-09T00:05:49.109306+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:05:49.109306+0000","last_undegraded":"2024-03-09T00:05:49.109306+0000","last_fullsized":"2024-03-09T00:05:49.109306+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":149,"num_read_kb":136,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050774+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:05:05.050774+0000","last_peered":"2024-03-09T00:05:05.050774+0000","last_clean":"2024-03-09T00:05:05.050774+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:05:05.050774+0000","last_undegraded":"2024-03-09T00:05:05.050774+0000","last_fullsized":"2024-03-09T00:05:05.050774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054832+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:05:05.054832+0000","last_peered":"2024-03-09T00:05:05.054832+0000","last_clean":"2024-03-09T00:05:05.054832+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:05:05.054832+0000","last_undegraded":"2024-03-09T00:05:05.054832+0000","last_fullsized":"2024-03-09T00:05:05.054832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053332+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:05:05.053332+0000","last_peered":"2024-03-09T00:05:05.053332+0000","last_clean":"2024-03-09T00:05:05.053332+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:05:05.053332+0000","last_undegraded":"2024-03-09T00:05:05.053332+0000","last_fullsized":"2024-03-09T00:05:05.053332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052233+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:05:05.052233+0000","last_peered":"2024-03-09T00:05:05.052233+0000","last_clean":"2024-03-09T00:05:05.052233+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:05:05.052233+0000","last_undegraded":"2024-03-09T00:05:05.052233+0000","last_fullsized":"2024-03-09T00:05:05.052233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054963+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:05:05.054963+0000","last_peered":"2024-03-09T00:05:05.054963+0000","last_clean":"2024-03-09T00:05:05.054963+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:05:05.054963+0000","last_undegraded":"2024-03-09T00:05:05.054963+0000","last_fullsized":"2024-03-09T00:05:05.054963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053976+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:05:05.053976+0000","last_peered":"2024-03-09T00:05:05.053976+0000","last_clean":"2024-03-09T00:05:05.053976+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:05:05.053976+0000","last_undegraded":"2024-03-09T00:05:05.053976+0000","last_fullsized":"2024-03-09T00:05:05.053976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.059283+0000","last_change":"2024-03-09T00:05:05.059283+0000","last_active":"2024-03-09T00:05:05.059283+0000","last_peered":"2024-03-09T00:05:05.059283+0000","last_clean":"2024-03-09T00:05:05.059283+0000","last_became_active":"2024-03-09T00:05:05.058822+0000","last_became_peered":"2024-03-09T00:05:05.058822+0000","last_unstale":"2024-03-09T00:05:05.059283+0000","last_undegraded":"2024-03-09T00:05:05.059283+0000","last_fullsized":"2024-03-09T00:05:05.059283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:13:02.253810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051710+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:05:05.051710+0000","last_peered":"2024-03-09T00:05:05.051710+0000","last_clean":"2024-03-09T00:05:05.051710+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:05:05.051710+0000","last_undegraded":"2024-03-09T00:05:05.051710+0000","last_fullsized":"2024-03-09T00:05:05.051710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.137995+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:05:05.137995+0000","last_peered":"2024-03-09T00:05:05.137995+0000","last_clean":"2024-03-09T00:05:05.137995+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:05:05.137995+0000","last_undegraded":"2024-03-09T00:05:05.137995+0000","last_fullsized":"2024-03-09T00:05:05.137995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056740+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:05:05.056740+0000","last_peered":"2024-03-09T00:05:05.056740+0000","last_clean":"2024-03-09T00:05:05.056740+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:05:05.056740+0000","last_undegraded":"2024-03-09T00:05:05.056740+0000","last_fullsized":"2024-03-09T00:05:05.056740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052918+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:05:05.052918+0000","last_peered":"2024-03-09T00:05:05.052918+0000","last_clean":"2024-03-09T00:05:05.052918+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:05:05.052918+0000","last_undegraded":"2024-03-09T00:05:05.052918+0000","last_fullsized":"2024-03-09T00:05:05.052918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049721+0000","last_change":"2024-03-09T00:04:13.330355+0000","last_active":"2024-03-09T00:05:05.049721+0000","last_peered":"2024-03-09T00:05:05.049721+0000","last_clean":"2024-03-09T00:05:05.049721+0000","last_became_active":"2024-03-09T00:04:13.330012+0000","last_became_peered":"2024-03-09T00:04:13.330012+0000","last_unstale":"2024-03-09T00:05:05.049721+0000","last_undegraded":"2024-03-09T00:05:05.049721+0000","last_fullsized":"2024-03-09T00:05:05.049721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:46.699807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053962+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:05:05.053962+0000","last_peered":"2024-03-09T00:05:05.053962+0000","last_clean":"2024-03-09T00:05:05.053962+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:05:05.053962+0000","last_undegraded":"2024-03-09T00:05:05.053962+0000","last_fullsized":"2024-03-09T00:05:05.053962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052971+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:05:05.052971+0000","last_peered":"2024-03-09T00:05:05.052971+0000","last_clean":"2024-03-09T00:05:05.052971+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:05:05.052971+0000","last_undegraded":"2024-03-09T00:05:05.052971+0000","last_fullsized":"2024-03-09T00:05:05.052971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051325+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:05:05.051325+0000","last_peered":"2024-03-09T00:05:05.051325+0000","last_clean":"2024-03-09T00:05:05.051325+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:05:05.051325+0000","last_undegraded":"2024-03-09T00:05:05.051325+0000","last_fullsized":"2024-03-09T00:05:05.051325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138598+0000","last_change":"2024-03-09T00:05:05.138598+0000","last_active":"2024-03-09T00:05:05.138598+0000","last_peered":"2024-03-09T00:05:05.138598+0000","last_clean":"2024-03-09T00:05:05.138598+0000","last_became_active":"2024-03-09T00:05:05.138166+0000","last_became_peered":"2024-03-09T00:05:05.138166+0000","last_unstale":"2024-03-09T00:05:05.138598+0000","last_undegraded":"2024-03-09T00:05:05.138598+0000","last_fullsized":"2024-03-09T00:05:05.138598+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:59:42.864722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050053+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:05:05.050053+0000","last_peered":"2024-03-09T00:05:05.050053+0000","last_clean":"2024-03-09T00:05:05.050053+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:05:05.050053+0000","last_undegraded":"2024-03-09T00:05:05.050053+0000","last_fullsized":"2024-03-09T00:05:05.050053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051497+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:05:05.051497+0000","last_peered":"2024-03-09T00:05:05.051497+0000","last_clean":"2024-03-09T00:05:05.051497+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:05:05.051497+0000","last_undegraded":"2024-03-09T00:05:05.051497+0000","last_fullsized":"2024-03-09T00:05:05.051497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053406+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:05:05.053406+0000","last_peered":"2024-03-09T00:05:05.053406+0000","last_clean":"2024-03-09T00:05:05.053406+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:05:05.053406+0000","last_undegraded":"2024-03-09T00:05:05.053406+0000","last_fullsized":"2024-03-09T00:05:05.053406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052732+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:05:05.052732+0000","last_peered":"2024-03-09T00:05:05.052732+0000","last_clean":"2024-03-09T00:05:05.052732+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:05:05.052732+0000","last_undegraded":"2024-03-09T00:05:05.052732+0000","last_fullsized":"2024-03-09T00:05:05.052732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057196+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:05:05.057196+0000","last_peered":"2024-03-09T00:05:05.057196+0000","last_clean":"2024-03-09T00:05:05.057196+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:05:05.057196+0000","last_undegraded":"2024-03-09T00:05:05.057196+0000","last_fullsized":"2024-03-09T00:05:05.057196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053098+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:05:05.053098+0000","last_peered":"2024-03-09T00:05:05.053098+0000","last_clean":"2024-03-09T00:05:05.053098+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:05:05.053098+0000","last_undegraded":"2024-03-09T00:05:05.053098+0000","last_fullsized":"2024-03-09T00:05:05.053098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138164+0000","last_change":"2024-03-09T00:05:05.138164+0000","last_active":"2024-03-09T00:05:05.138164+0000","last_peered":"2024-03-09T00:05:05.138164+0000","last_clean":"2024-03-09T00:05:05.138164+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:05:05.138164+0000","last_undegraded":"2024-03-09T00:05:05.138164+0000","last_fullsized":"2024-03-09T00:05:05.138164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:12:44.310276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053931+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:05:05.053931+0000","last_peered":"2024-03-09T00:05:05.053931+0000","last_clean":"2024-03-09T00:05:05.053931+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:05:05.053931+0000","last_undegraded":"2024-03-09T00:05:05.053931+0000","last_fullsized":"2024-03-09T00:05:05.053931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052127+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:05:05.052127+0000","last_peered":"2024-03-09T00:05:05.052127+0000","last_clean":"2024-03-09T00:05:05.052127+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:05:05.052127+0000","last_undegraded":"2024-03-09T00:05:05.052127+0000","last_fullsized":"2024-03-09T00:05:05.052127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056336+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:05:05.056336+0000","last_peered":"2024-03-09T00:05:05.056336+0000","last_clean":"2024-03-09T00:05:05.056336+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:05:05.056336+0000","last_undegraded":"2024-03-09T00:05:05.056336+0000","last_fullsized":"2024-03-09T00:05:05.056336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058661+0000","last_change":"2024-03-09T00:05:05.058253+0000","last_active":"2024-03-09T00:05:05.058661+0000","last_peered":"2024-03-09T00:05:05.058661+0000","last_clean":"2024-03-09T00:05:05.058661+0000","last_became_active":"2024-03-09T00:05:05.057864+0000","last_became_peered":"2024-03-09T00:05:05.057864+0000","last_unstale":"2024-03-09T00:05:05.058661+0000","last_undegraded":"2024-03-09T00:05:05.058661+0000","last_fullsized":"2024-03-09T00:05:05.058661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:33:14.993355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138457+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:05:05.138457+0000","last_peered":"2024-03-09T00:05:05.138457+0000","last_clean":"2024-03-09T00:05:05.138457+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:05:05.138457+0000","last_undegraded":"2024-03-09T00:05:05.138457+0000","last_fullsized":"2024-03-09T00:05:05.138457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056546+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:05:05.056546+0000","last_peered":"2024-03-09T00:05:05.056546+0000","last_clean":"2024-03-09T00:05:05.056546+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:05:05.056546+0000","last_undegraded":"2024-03-09T00:05:05.056546+0000","last_fullsized":"2024-03-09T00:05:05.056546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053713+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:05:05.053713+0000","last_peered":"2024-03-09T00:05:05.053713+0000","last_clean":"2024-03-09T00:05:05.053713+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:05:05.053713+0000","last_undegraded":"2024-03-09T00:05:05.053713+0000","last_fullsized":"2024-03-09T00:05:05.053713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052102+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:05:05.052102+0000","last_peered":"2024-03-09T00:05:05.052102+0000","last_clean":"2024-03-09T00:05:05.052102+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:05:05.052102+0000","last_undegraded":"2024-03-09T00:05:05.052102+0000","last_fullsized":"2024-03-09T00:05:05.052102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056615+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:05:05.056615+0000","last_peered":"2024-03-09T00:05:05.056615+0000","last_clean":"2024-03-09T00:05:05.056615+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:05:05.056615+0000","last_undegraded":"2024-03-09T00:05:05.056615+0000","last_fullsized":"2024-03-09T00:05:05.056615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054833+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:05:05.054833+0000","last_peered":"2024-03-09T00:05:05.054833+0000","last_clean":"2024-03-09T00:05:05.054833+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:05:05.054833+0000","last_undegraded":"2024-03-09T00:05:05.054833+0000","last_fullsized":"2024-03-09T00:05:05.054833+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050878+0000","last_change":"2024-03-09T00:04:13.331922+0000","last_active":"2024-03-09T00:05:05.050878+0000","last_peered":"2024-03-09T00:05:05.050878+0000","last_clean":"2024-03-09T00:05:05.050878+0000","last_became_active":"2024-03-09T00:04:13.331610+0000","last_became_peered":"2024-03-09T00:04:13.331610+0000","last_unstale":"2024-03-09T00:05:05.050878+0000","last_undegraded":"2024-03-09T00:05:05.050878+0000","last_fullsized":"2024-03-09T00:05:05.050878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:29.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052861+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:05:05.052861+0000","last_peered":"2024-03-09T00:05:05.052861+0000","last_clean":"2024-03-09T00:05:05.052861+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:05:05.052861+0000","last_undegraded":"2024-03-09T00:05:05.052861+0000","last_fullsized":"2024-03-09T00:05:05.052861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049169+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:05:05.049169+0000","last_peered":"2024-03-09T00:05:05.049169+0000","last_clean":"2024-03-09T00:05:05.049169+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:05:05.049169+0000","last_undegraded":"2024-03-09T00:05:05.049169+0000","last_fullsized":"2024-03-09T00:05:05.049169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051655+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:05:05.051655+0000","last_peered":"2024-03-09T00:05:05.051655+0000","last_clean":"2024-03-09T00:05:05.051655+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:05:05.051655+0000","last_undegraded":"2024-03-09T00:05:05.051655+0000","last_fullsized":"2024-03-09T00:05:05.051655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049979+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:05:05.049979+0000","last_peered":"2024-03-09T00:05:05.049979+0000","last_clean":"2024-03-09T00:05:05.049979+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:05:05.049979+0000","last_undegraded":"2024-03-09T00:05:05.049979+0000","last_fullsized":"2024-03-09T00:05:05.049979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052356+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:05:05.052356+0000","last_peered":"2024-03-09T00:05:05.052356+0000","last_clean":"2024-03-09T00:05:05.052356+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:05:05.052356+0000","last_undegraded":"2024-03-09T00:05:05.052356+0000","last_fullsized":"2024-03-09T00:05:05.052356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138751+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:05:05.138751+0000","last_peered":"2024-03-09T00:05:05.138751+0000","last_clean":"2024-03-09T00:05:05.138751+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:05:05.138751+0000","last_undegraded":"2024-03-09T00:05:05.138751+0000","last_fullsized":"2024-03-09T00:05:05.138751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136810+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:05:05.136810+0000","last_peered":"2024-03-09T00:05:05.136810+0000","last_clean":"2024-03-09T00:05:05.136810+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:05:05.136810+0000","last_undegraded":"2024-03-09T00:05:05.136810+0000","last_fullsized":"2024-03-09T00:05:05.136810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138501+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:05:05.138501+0000","last_peered":"2024-03-09T00:05:05.138501+0000","last_clean":"2024-03-09T00:05:05.138501+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:05:05.138501+0000","last_undegraded":"2024-03-09T00:05:05.138501+0000","last_fullsized":"2024-03-09T00:05:05.138501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055772+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:05:05.055772+0000","last_peered":"2024-03-09T00:05:05.055772+0000","last_clean":"2024-03-09T00:05:05.055772+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:05:05.055772+0000","last_undegraded":"2024-03-09T00:05:05.055772+0000","last_fullsized":"2024-03-09T00:05:05.055772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049781+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:05:05.049781+0000","last_peered":"2024-03-09T00:05:05.049781+0000","last_clean":"2024-03-09T00:05:05.049781+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:05:05.049781+0000","last_undegraded":"2024-03-09T00:05:05.049781+0000","last_fullsized":"2024-03-09T00:05:05.049781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051777+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:05:05.051777+0000","last_peered":"2024-03-09T00:05:05.051777+0000","last_clean":"2024-03-09T00:05:05.051777+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:05:05.051777+0000","last_undegraded":"2024-03-09T00:05:05.051777+0000","last_fullsized":"2024-03-09T00:05:05.051777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049358+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:05:05.049358+0000","last_peered":"2024-03-09T00:05:05.049358+0000","last_clean":"2024-03-09T00:05:05.049358+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:05:05.049358+0000","last_undegraded":"2024-03-09T00:05:05.049358+0000","last_fullsized":"2024-03-09T00:05:05.049358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054636+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:05:05.054636+0000","last_peered":"2024-03-09T00:05:05.054636+0000","last_clean":"2024-03-09T00:05:05.054636+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:05:05.054636+0000","last_undegraded":"2024-03-09T00:05:05.054636+0000","last_fullsized":"2024-03-09T00:05:05.054636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138261+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:05:05.138261+0000","last_peered":"2024-03-09T00:05:05.138261+0000","last_clean":"2024-03-09T00:05:05.138261+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:05:05.138261+0000","last_undegraded":"2024-03-09T00:05:05.138261+0000","last_fullsized":"2024-03-09T00:05:05.138261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051370+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:05:05.051370+0000","last_peered":"2024-03-09T00:05:05.051370+0000","last_clean":"2024-03-09T00:05:05.051370+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:05:05.051370+0000","last_undegraded":"2024-03-09T00:05:05.051370+0000","last_fullsized":"2024-03-09T00:05:05.051370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049312+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:05:05.049312+0000","last_peered":"2024-03-09T00:05:05.049312+0000","last_clean":"2024-03-09T00:05:05.049312+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:05:05.049312+0000","last_undegraded":"2024-03-09T00:05:05.049312+0000","last_fullsized":"2024-03-09T00:05:05.049312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138730+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:05:05.138730+0000","last_peered":"2024-03-09T00:05:05.138730+0000","last_clean":"2024-03-09T00:05:05.138730+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:05:05.138730+0000","last_undegraded":"2024-03-09T00:05:05.138730+0000","last_fullsized":"2024-03-09T00:05:05.138730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050211+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:05:05.050211+0000","last_peered":"2024-03-09T00:05:05.050211+0000","last_clean":"2024-03-09T00:05:05.050211+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:05:05.050211+0000","last_undegraded":"2024-03-09T00:05:05.050211+0000","last_fullsized":"2024-03-09T00:05:05.050211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057000+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:05:05.057000+0000","last_peered":"2024-03-09T00:05:05.057000+0000","last_clean":"2024-03-09T00:05:05.057000+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:05:05.057000+0000","last_undegraded":"2024-03-09T00:05:05.057000+0000","last_fullsized":"2024-03-09T00:05:05.057000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055823+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:05:05.055823+0000","last_peered":"2024-03-09T00:05:05.055823+0000","last_clean":"2024-03-09T00:05:05.055823+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:05:05.055823+0000","last_undegraded":"2024-03-09T00:05:05.055823+0000","last_fullsized":"2024-03-09T00:05:05.055823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053442+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:05:05.053442+0000","last_peered":"2024-03-09T00:05:05.053442+0000","last_clean":"2024-03-09T00:05:05.053442+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:05:05.053442+0000","last_undegraded":"2024-03-09T00:05:05.053442+0000","last_fullsized":"2024-03-09T00:05:05.053442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056696+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:05:05.056696+0000","last_peered":"2024-03-09T00:05:05.056696+0000","last_clean":"2024-03-09T00:05:05.056696+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:05:05.056696+0000","last_undegraded":"2024-03-09T00:05:05.056696+0000","last_fullsized":"2024-03-09T00:05:05.056696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136421+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:05:05.136421+0000","last_peered":"2024-03-09T00:05:05.136421+0000","last_clean":"2024-03-09T00:05:05.136421+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:05:05.136421+0000","last_undegraded":"2024-03-09T00:05:05.136421+0000","last_fullsized":"2024-03-09T00:05:05.136421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054593+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:05:05.054593+0000","last_peered":"2024-03-09T00:05:05.054593+0000","last_clean":"2024-03-09T00:05:05.054593+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:05:05.054593+0000","last_undegraded":"2024-03-09T00:05:05.054593+0000","last_fullsized":"2024-03-09T00:05:05.054593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057094+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:05:05.057094+0000","last_peered":"2024-03-09T00:05:05.057094+0000","last_clean":"2024-03-09T00:05:05.057094+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:05:05.057094+0000","last_undegraded":"2024-03-09T00:05:05.057094+0000","last_fullsized":"2024-03-09T00:05:05.057094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053282+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:05:05.053282+0000","last_peered":"2024-03-09T00:05:05.053282+0000","last_clean":"2024-03-09T00:05:05.053282+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:05:05.053282+0000","last_undegraded":"2024-03-09T00:05:05.053282+0000","last_fullsized":"2024-03-09T00:05:05.053282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053043+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:05:05.053043+0000","last_peered":"2024-03-09T00:05:05.053043+0000","last_clean":"2024-03-09T00:05:05.053043+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:05:05.053043+0000","last_undegraded":"2024-03-09T00:05:05.053043+0000","last_fullsized":"2024-03-09T00:05:05.053043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057584+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:05:05.057584+0000","last_peered":"2024-03-09T00:05:05.057584+0000","last_clean":"2024-03-09T00:05:05.057584+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:05:05.057584+0000","last_undegraded":"2024-03-09T00:05:05.057584+0000","last_fullsized":"2024-03-09T00:05:05.057584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054743+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:05:05.054743+0000","last_peered":"2024-03-09T00:05:05.054743+0000","last_clean":"2024-03-09T00:05:05.054743+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:05:05.054743+0000","last_undegraded":"2024-03-09T00:05:05.054743+0000","last_fullsized":"2024-03-09T00:05:05.054743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052584+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:05:05.052584+0000","last_peered":"2024-03-09T00:05:05.052584+0000","last_clean":"2024-03-09T00:05:05.052584+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:05:05.052584+0000","last_undegraded":"2024-03-09T00:05:05.052584+0000","last_fullsized":"2024-03-09T00:05:05.052584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051228+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:05:05.051228+0000","last_peered":"2024-03-09T00:05:05.051228+0000","last_clean":"2024-03-09T00:05:05.051228+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:05:05.051228+0000","last_undegraded":"2024-03-09T00:05:05.051228+0000","last_fullsized":"2024-03-09T00:05:05.051228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136717+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:05:05.136717+0000","last_peered":"2024-03-09T00:05:05.136717+0000","last_clean":"2024-03-09T00:05:05.136717+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:05:05.136717+0000","last_undegraded":"2024-03-09T00:05:05.136717+0000","last_fullsized":"2024-03-09T00:05:05.136717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052834+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:05:05.052834+0000","last_peered":"2024-03-09T00:05:05.052834+0000","last_clean":"2024-03-09T00:05:05.052834+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:05:05.052834+0000","last_undegraded":"2024-03-09T00:05:05.052834+0000","last_fullsized":"2024-03-09T00:05:05.052834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055931+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:05:05.055931+0000","last_peered":"2024-03-09T00:05:05.055931+0000","last_clean":"2024-03-09T00:05:05.055931+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:05:05.055931+0000","last_undegraded":"2024-03-09T00:05:05.055931+0000","last_fullsized":"2024-03-09T00:05:05.055931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051815+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:05:05.051815+0000","last_peered":"2024-03-09T00:05:05.051815+0000","last_clean":"2024-03-09T00:05:05.051815+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:05:05.051815+0000","last_undegraded":"2024-03-09T00:05:05.051815+0000","last_fullsized":"2024-03-09T00:05:05.051815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050101+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:05:05.050101+0000","last_peered":"2024-03-09T00:05:05.050101+0000","last_clean":"2024-03-09T00:05:05.050101+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:05:05.050101+0000","last_undegraded":"2024-03-09T00:05:05.050101+0000","last_fullsized":"2024-03-09T00:05:05.050101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053808+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:05:05.053808+0000","last_peered":"2024-03-09T00:05:05.053808+0000","last_clean":"2024-03-09T00:05:05.053808+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:05:05.053808+0000","last_undegraded":"2024-03-09T00:05:05.053808+0000","last_fullsized":"2024-03-09T00:05:05.053808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049386+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:05:05.049386+0000","last_peered":"2024-03-09T00:05:05.049386+0000","last_clean":"2024-03-09T00:05:05.049386+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:05:05.049386+0000","last_undegraded":"2024-03-09T00:05:05.049386+0000","last_fullsized":"2024-03-09T00:05:05.049386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056305+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:05:05.056305+0000","last_peered":"2024-03-09T00:05:05.056305+0000","last_clean":"2024-03-09T00:05:05.056305+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:05:05.056305+0000","last_undegraded":"2024-03-09T00:05:05.056305+0000","last_fullsized":"2024-03-09T00:05:05.056305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052606+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:05:05.052606+0000","last_peered":"2024-03-09T00:05:05.052606+0000","last_clean":"2024-03-09T00:05:05.052606+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:05:05.052606+0000","last_undegraded":"2024-03-09T00:05:05.052606+0000","last_fullsized":"2024-03-09T00:05:05.052606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055098+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:05:05.055098+0000","last_peered":"2024-03-09T00:05:05.055098+0000","last_clean":"2024-03-09T00:05:05.055098+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:05:05.055098+0000","last_undegraded":"2024-03-09T00:05:05.055098+0000","last_fullsized":"2024-03-09T00:05:05.055098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":155,"num_read_kb":142,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":192462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383714,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":500249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645351,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712312,"statfs":{"total":95995035648,"available":95961407488,"internally_reserved":0,"allocated":1695744,"data_stored":1150071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808876,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":504927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070514,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":500853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332150,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1622016,"data_stored":1095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495676,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574933,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":559509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411408,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":559543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182474,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":496257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920838,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1024000,"data_stored":500271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757312,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1626112,"data_stored":1090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:05:56.153 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph pg dump --format=json 2024-03-09T00:05:56.393 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:05:57.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:57 smithi073 ceph-mon[32309]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:05:57.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:57 smithi073 ceph-mon[32309]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:57.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:57 smithi179 ceph-mon[36637]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:05:57.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:57 smithi179 ceph-mon[36637]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:57 smithi102 ceph-mon[36106]: from='client.15120 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:05:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:57 smithi102 ceph-mon[36106]: pgmap v347: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 -- 172.21.15.73:0/1785048052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 msgr2=0x7efc94100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 --2- 172.21.15.73:0/1785048052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94100d60 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b0d0 tx=0x7efc8c00b3e0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 -- 172.21.15.73:0/1785048052 shutdown_connections 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 --2- 172.21.15.73:0/1785048052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94100d60 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 --2- 172.21.15.73:0/1785048052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 0x7efc9410ed20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 --2- 172.21.15.73:0/1785048052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 0x7efc94106a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.247+0000 7efc9bd98700 1 -- 172.21.15.73:0/1785048052 >> 172.21.15.73:0/1785048052 conn(0x7efc94076020 msgr2=0x7efc94076430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.248+0000 7efc9bd98700 1 -- 172.21.15.73:0/1785048052 shutdown_connections 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.248+0000 7efc9bd98700 1 -- 172.21.15.73:0/1785048052 wait complete. 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.248+0000 7efc9bd98700 1 Processor -- start 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.248+0000 7efc9bd98700 1 -- start start 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 0x7efc94101e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 0x7efc94102380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94112150 con 0x7efc94100980 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc94111fd0 con 0x7efc941071a0 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9bd98700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc941122d0 con 0x7efc94101330 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:58.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43930/0 (socket says 172.21.15.73:43930) 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 -- 172.21.15.73:0/2573244756 learned_addr learned my addr 172.21.15.73:0/2573244756 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99333700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 0x7efc94101e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc9a335700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 0x7efc94102380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 -- 172.21.15.73:0/2573244756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 msgr2=0x7efc94101e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 0x7efc94101e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 -- 172.21.15.73:0/2573244756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 msgr2=0x7efc94102380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 0x7efc94102380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 -- 172.21.15.73:0/2573244756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc941a6af0 con 0x7efc94100980 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc99b34700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b820 tx=0x7efc8c004db0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.249+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c021020 con 0x7efc94100980 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.250+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc941a6d80 con 0x7efc94100980 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.250+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efc8c007cc0 con 0x7efc94100980 2024-03-09T00:05:58.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.250+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efc8c005280 con 0x7efc94100980 2024-03-09T00:05:58.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.250+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc941a71d0 con 0x7efc94100980 2024-03-09T00:05:58.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.251+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc78005330 con 0x7efc94100980 2024-03-09T00:05:58.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.252+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7efc8c024020 con 0x7efc94100980 2024-03-09T00:05:58.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.253+0000 7efc8affd700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 0x7efc800a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:05:58.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.253+0000 7efc99333700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 0x7efc800a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:05:58.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.253+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7efc8c00f020 con 0x7efc94100980 2024-03-09T00:05:58.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.254+0000 7efc99333700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 0x7efc800a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc94108680 tx=0x7efc8400b3c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:05:58.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.257+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7efc8c09d5f0 con 0x7efc94100980 2024-03-09T00:05:58.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.451+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7efc78000bc0 con 0x7efc800a2920 2024-03-09T00:05:58.455 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.460+0000 7efc8affd700 1 -- 172.21.15.73:0/2573244756 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340270 (secure 0 0 0) 0x7efc78000bc0 con 0x7efc800a2920 2024-03-09T00:05:58.460 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.463+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 msgr2=0x7efc800a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 0x7efc800a4de0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7efc94108680 tx=0x7efc8400b3c0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 msgr2=0x7efc94101900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7efc8c00b820 tx=0x7efc8c004db0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 shutdown_connections 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efc94100980 0x7efc94101900 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7efc800a2920 0x7efc800a4de0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efc941071a0 0x7efc94102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 --2- 172.21.15.73:0/2573244756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efc94101330 0x7efc94101e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:05:58.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 >> 172.21.15.73:0/2573244756 conn(0x7efc94076020 msgr2=0x7efc940fe790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:05:58.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 shutdown_connections 2024-03-09T00:05:58.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:05:58.464+0000 7efc9bd98700 1 -- 172.21.15.73:0/2573244756 wait complete. 2024-03-09T00:05:58.464 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:05:59.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:05:59 smithi179 ceph-mon[36637]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:59.365 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:05:59 smithi073 ceph-mon[32309]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:59.396 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":348,"stamp":"2024-03-09T00:05:57.955883+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":654756,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":301,"num_read_kb":266,"num_write":262,"num_write_kb":3030,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":207,"ondisk_log_size":207,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":388140,"kb_used_data":13952,"kb_used_omap":18,"kb_used_meta":373997,"kb_avail":1124553684,"statfs":{"total":1151940427776,"available":1151542972416,"internally_reserved":0,"allocated":14286848,"data_stored":7950170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":382973241},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":20,"num_read_kb":10,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002857"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055886+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:05:05.055886+0000","last_peered":"2024-03-09T00:05:05.055886+0000","last_clean":"2024-03-09T00:05:05.055886+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:05:05.055886+0000","last_undegraded":"2024-03-09T00:05:05.055886+0000","last_fullsized":"2024-03-09T00:05:05.055886+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136779+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:05:05.136779+0000","last_peered":"2024-03-09T00:05:05.136779+0000","last_clean":"2024-03-09T00:05:05.136779+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:05:05.136779+0000","last_undegraded":"2024-03-09T00:05:05.136779+0000","last_fullsized":"2024-03-09T00:05:05.136779+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136593+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:05:05.136593+0000","last_peered":"2024-03-09T00:05:05.136593+0000","last_clean":"2024-03-09T00:05:05.136593+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:05:05.136593+0000","last_undegraded":"2024-03-09T00:05:05.136593+0000","last_fullsized":"2024-03-09T00:05:05.136593+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138656+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:05:05.138656+0000","last_peered":"2024-03-09T00:05:05.138656+0000","last_clean":"2024-03-09T00:05:05.138656+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:05:05.138656+0000","last_undegraded":"2024-03-09T00:05:05.138656+0000","last_fullsized":"2024-03-09T00:05:05.138656+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050159+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:05:05.050159+0000","last_peered":"2024-03-09T00:05:05.050159+0000","last_clean":"2024-03-09T00:05:05.050159+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:05:05.050159+0000","last_undegraded":"2024-03-09T00:05:05.050159+0000","last_fullsized":"2024-03-09T00:05:05.050159+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":31,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058033+0000","last_change":"2024-03-09T00:05:05.058033+0000","last_active":"2024-03-09T00:05:05.058033+0000","last_peered":"2024-03-09T00:05:05.058033+0000","last_clean":"2024-03-09T00:05:05.058033+0000","last_became_active":"2024-03-09T00:05:05.057575+0000","last_became_peered":"2024-03-09T00:05:05.057575+0000","last_unstale":"2024-03-09T00:05:05.058033+0000","last_undegraded":"2024-03-09T00:05:05.058033+0000","last_fullsized":"2024-03-09T00:05:05.058033+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:35:58.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056728+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:05:05.056728+0000","last_peered":"2024-03-09T00:05:05.056728+0000","last_clean":"2024-03-09T00:05:05.056728+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:05:05.056728+0000","last_undegraded":"2024-03-09T00:05:05.056728+0000","last_fullsized":"2024-03-09T00:05:05.056728+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049165+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:05:05.049165+0000","last_peered":"2024-03-09T00:05:05.049165+0000","last_clean":"2024-03-09T00:05:05.049165+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:05:05.049165+0000","last_undegraded":"2024-03-09T00:05:05.049165+0000","last_fullsized":"2024-03-09T00:05:05.049165+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056946+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:05:05.056946+0000","last_peered":"2024-03-09T00:05:05.056946+0000","last_clean":"2024-03-09T00:05:05.056946+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:05:05.056946+0000","last_undegraded":"2024-03-09T00:05:05.056946+0000","last_fullsized":"2024-03-09T00:05:05.056946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054029+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:05:05.054029+0000","last_peered":"2024-03-09T00:05:05.054029+0000","last_clean":"2024-03-09T00:05:05.054029+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:05:05.054029+0000","last_undegraded":"2024-03-09T00:05:05.054029+0000","last_fullsized":"2024-03-09T00:05:05.054029+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056580+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:05:05.056580+0000","last_peered":"2024-03-09T00:05:05.056580+0000","last_clean":"2024-03-09T00:05:05.056580+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:05:05.056580+0000","last_undegraded":"2024-03-09T00:05:05.056580+0000","last_fullsized":"2024-03-09T00:05:05.056580+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138418+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:05:05.138418+0000","last_peered":"2024-03-09T00:05:05.138418+0000","last_clean":"2024-03-09T00:05:05.138418+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:05:05.138418+0000","last_undegraded":"2024-03-09T00:05:05.138418+0000","last_fullsized":"2024-03-09T00:05:05.138418+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052287+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:05:05.052287+0000","last_peered":"2024-03-09T00:05:05.052287+0000","last_clean":"2024-03-09T00:05:05.052287+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:05:05.052287+0000","last_undegraded":"2024-03-09T00:05:05.052287+0000","last_fullsized":"2024-03-09T00:05:05.052287+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136435+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:05:05.136435+0000","last_peered":"2024-03-09T00:05:05.136435+0000","last_clean":"2024-03-09T00:05:05.136435+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:05:05.136435+0000","last_undegraded":"2024-03-09T00:05:05.136435+0000","last_fullsized":"2024-03-09T00:05:05.136435+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136587+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:05:05.136587+0000","last_peered":"2024-03-09T00:05:05.136587+0000","last_clean":"2024-03-09T00:05:05.136587+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:05:05.136587+0000","last_undegraded":"2024-03-09T00:05:05.136587+0000","last_fullsized":"2024-03-09T00:05:05.136587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055637+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:05:05.055637+0000","last_peered":"2024-03-09T00:05:05.055637+0000","last_clean":"2024-03-09T00:05:05.055637+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:05:05.055637+0000","last_undegraded":"2024-03-09T00:05:05.055637+0000","last_fullsized":"2024-03-09T00:05:05.055637+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055631+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:05:05.055631+0000","last_peered":"2024-03-09T00:05:05.055631+0000","last_clean":"2024-03-09T00:05:05.055631+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:05:05.055631+0000","last_undegraded":"2024-03-09T00:05:05.055631+0000","last_fullsized":"2024-03-09T00:05:05.055631+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051647+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:05:05.051647+0000","last_peered":"2024-03-09T00:05:05.051647+0000","last_clean":"2024-03-09T00:05:05.051647+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:05:05.051647+0000","last_undegraded":"2024-03-09T00:05:05.051647+0000","last_fullsized":"2024-03-09T00:05:05.051647+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053008+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:05:05.053008+0000","last_peered":"2024-03-09T00:05:05.053008+0000","last_clean":"2024-03-09T00:05:05.053008+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:05:05.053008+0000","last_undegraded":"2024-03-09T00:05:05.053008+0000","last_fullsized":"2024-03-09T00:05:05.053008+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138220+0000","last_change":"2024-03-09T00:05:05.138220+0000","last_active":"2024-03-09T00:05:05.138220+0000","last_peered":"2024-03-09T00:05:05.138220+0000","last_clean":"2024-03-09T00:05:05.138220+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:05:05.138220+0000","last_undegraded":"2024-03-09T00:05:05.138220+0000","last_fullsized":"2024-03-09T00:05:05.138220+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:41.455601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":32,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051852+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:05:05.051852+0000","last_peered":"2024-03-09T00:05:05.051852+0000","last_clean":"2024-03-09T00:05:05.051852+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:05:05.051852+0000","last_undegraded":"2024-03-09T00:05:05.051852+0000","last_fullsized":"2024-03-09T00:05:05.051852+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049674+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:05:05.049674+0000","last_peered":"2024-03-09T00:05:05.049674+0000","last_clean":"2024-03-09T00:05:05.049674+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:05:05.049674+0000","last_undegraded":"2024-03-09T00:05:05.049674+0000","last_fullsized":"2024-03-09T00:05:05.049674+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.137818+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:05:05.137818+0000","last_peered":"2024-03-09T00:05:05.137818+0000","last_clean":"2024-03-09T00:05:05.137818+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:05:05.137818+0000","last_undegraded":"2024-03-09T00:05:05.137818+0000","last_fullsized":"2024-03-09T00:05:05.137818+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049925+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:05:05.049925+0000","last_peered":"2024-03-09T00:05:05.049925+0000","last_clean":"2024-03-09T00:05:05.049925+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:05:05.049925+0000","last_undegraded":"2024-03-09T00:05:05.049925+0000","last_fullsized":"2024-03-09T00:05:05.049925+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136845+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:05:05.136845+0000","last_peered":"2024-03-09T00:05:05.136845+0000","last_clean":"2024-03-09T00:05:05.136845+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:05:05.136845+0000","last_undegraded":"2024-03-09T00:05:05.136845+0000","last_fullsized":"2024-03-09T00:05:05.136845+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138164+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:05:05.138164+0000","last_peered":"2024-03-09T00:05:05.138164+0000","last_clean":"2024-03-09T00:05:05.138164+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:05:05.138164+0000","last_undegraded":"2024-03-09T00:05:05.138164+0000","last_fullsized":"2024-03-09T00:05:05.138164+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":39,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.142560+0000","last_change":"2024-03-09T00:05:05.142560+0000","last_active":"2024-03-09T00:05:05.142560+0000","last_peered":"2024-03-09T00:05:05.142560+0000","last_clean":"2024-03-09T00:05:05.142560+0000","last_became_active":"2024-03-09T00:05:05.139877+0000","last_became_peered":"2024-03-09T00:05:05.139877+0000","last_unstale":"2024-03-09T00:05:05.142560+0000","last_undegraded":"2024-03-09T00:05:05.142560+0000","last_fullsized":"2024-03-09T00:05:05.142560+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:30.441723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056923+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:05:05.056923+0000","last_peered":"2024-03-09T00:05:05.056923+0000","last_clean":"2024-03-09T00:05:05.056923+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:05:05.056923+0000","last_undegraded":"2024-03-09T00:05:05.056923+0000","last_fullsized":"2024-03-09T00:05:05.056923+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138277+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:05:05.138277+0000","last_peered":"2024-03-09T00:05:05.138277+0000","last_clean":"2024-03-09T00:05:05.138277+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:05:05.138277+0000","last_undegraded":"2024-03-09T00:05:05.138277+0000","last_fullsized":"2024-03-09T00:05:05.138277+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050427+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:05:05.050427+0000","last_peered":"2024-03-09T00:05:05.050427+0000","last_clean":"2024-03-09T00:05:05.050427+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:05:05.050427+0000","last_undegraded":"2024-03-09T00:05:05.050427+0000","last_fullsized":"2024-03-09T00:05:05.050427+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054482+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:05:05.054482+0000","last_peered":"2024-03-09T00:05:05.054482+0000","last_clean":"2024-03-09T00:05:05.054482+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:05:05.054482+0000","last_undegraded":"2024-03-09T00:05:05.054482+0000","last_fullsized":"2024-03-09T00:05:05.054482+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052692+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:05:05.052692+0000","last_peered":"2024-03-09T00:05:05.052692+0000","last_clean":"2024-03-09T00:05:05.052692+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:05:05.052692+0000","last_undegraded":"2024-03-09T00:05:05.052692+0000","last_fullsized":"2024-03-09T00:05:05.052692+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054835+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:05:05.054835+0000","last_peered":"2024-03-09T00:05:05.054835+0000","last_clean":"2024-03-09T00:05:05.054835+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:05:05.054835+0000","last_undegraded":"2024-03-09T00:05:05.054835+0000","last_fullsized":"2024-03-09T00:05:05.054835+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049785+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:05:05.049785+0000","last_peered":"2024-03-09T00:05:05.049785+0000","last_clean":"2024-03-09T00:05:05.049785+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:05:05.049785+0000","last_undegraded":"2024-03-09T00:05:05.049785+0000","last_fullsized":"2024-03-09T00:05:05.049785+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052486+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:05:05.052486+0000","last_peered":"2024-03-09T00:05:05.052486+0000","last_clean":"2024-03-09T00:05:05.052486+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:05:05.052486+0000","last_undegraded":"2024-03-09T00:05:05.052486+0000","last_fullsized":"2024-03-09T00:05:05.052486+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053680+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:05:05.053680+0000","last_peered":"2024-03-09T00:05:05.053680+0000","last_clean":"2024-03-09T00:05:05.053680+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:05:05.053680+0000","last_undegraded":"2024-03-09T00:05:05.053680+0000","last_fullsized":"2024-03-09T00:05:05.053680+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056415+0000","last_change":"2024-03-09T00:05:05.056415+0000","last_active":"2024-03-09T00:05:05.056415+0000","last_peered":"2024-03-09T00:05:05.056415+0000","last_clean":"2024-03-09T00:05:05.056415+0000","last_became_active":"2024-03-09T00:05:05.056025+0000","last_became_peered":"2024-03-09T00:05:05.056025+0000","last_unstale":"2024-03-09T00:05:05.056415+0000","last_undegraded":"2024-03-09T00:05:05.056415+0000","last_fullsized":"2024-03-09T00:05:05.056415+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:29.970623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055504+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:05:05.055504+0000","last_peered":"2024-03-09T00:05:05.055504+0000","last_clean":"2024-03-09T00:05:05.055504+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:05:05.055504+0000","last_undegraded":"2024-03-09T00:05:05.055504+0000","last_fullsized":"2024-03-09T00:05:05.055504+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052237+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:05:05.052237+0000","last_peered":"2024-03-09T00:05:05.052237+0000","last_clean":"2024-03-09T00:05:05.052237+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:05:05.052237+0000","last_undegraded":"2024-03-09T00:05:05.052237+0000","last_fullsized":"2024-03-09T00:05:05.052237+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049824+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:05:05.049824+0000","last_peered":"2024-03-09T00:05:05.049824+0000","last_clean":"2024-03-09T00:05:05.049824+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:05:05.049824+0000","last_undegraded":"2024-03-09T00:05:05.049824+0000","last_fullsized":"2024-03-09T00:05:05.049824+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136618+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:05:05.136618+0000","last_peered":"2024-03-09T00:05:05.136618+0000","last_clean":"2024-03-09T00:05:05.136618+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:05:05.136618+0000","last_undegraded":"2024-03-09T00:05:05.136618+0000","last_fullsized":"2024-03-09T00:05:05.136618+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053698+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:05:05.053698+0000","last_peered":"2024-03-09T00:05:05.053698+0000","last_clean":"2024-03-09T00:05:05.053698+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:05:05.053698+0000","last_undegraded":"2024-03-09T00:05:05.053698+0000","last_fullsized":"2024-03-09T00:05:05.053698+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050966+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:05:05.050966+0000","last_peered":"2024-03-09T00:05:05.050966+0000","last_clean":"2024-03-09T00:05:05.050966+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:05:05.050966+0000","last_undegraded":"2024-03-09T00:05:05.050966+0000","last_fullsized":"2024-03-09T00:05:05.050966+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049265+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:05:05.049265+0000","last_peered":"2024-03-09T00:05:05.049265+0000","last_clean":"2024-03-09T00:05:05.049265+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:05:05.049265+0000","last_undegraded":"2024-03-09T00:05:05.049265+0000","last_fullsized":"2024-03-09T00:05:05.049265+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053532+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:05:05.053532+0000","last_peered":"2024-03-09T00:05:05.053532+0000","last_clean":"2024-03-09T00:05:05.053532+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:05:05.053532+0000","last_undegraded":"2024-03-09T00:05:05.053532+0000","last_fullsized":"2024-03-09T00:05:05.053532+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056957+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:05:05.056957+0000","last_peered":"2024-03-09T00:05:05.056957+0000","last_clean":"2024-03-09T00:05:05.056957+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:05:05.056957+0000","last_undegraded":"2024-03-09T00:05:05.056957+0000","last_fullsized":"2024-03-09T00:05:05.056957+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.140115+0000","last_change":"2024-03-09T00:05:05.139426+0000","last_active":"2024-03-09T00:05:05.140115+0000","last_peered":"2024-03-09T00:05:05.140115+0000","last_clean":"2024-03-09T00:05:05.140115+0000","last_became_active":"2024-03-09T00:05:05.138971+0000","last_became_peered":"2024-03-09T00:05:05.138971+0000","last_unstale":"2024-03-09T00:05:05.140115+0000","last_undegraded":"2024-03-09T00:05:05.140115+0000","last_fullsized":"2024-03-09T00:05:05.140115+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:40:59.870972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057274+0000","last_change":"2024-03-09T00:04:13.334152+0000","last_active":"2024-03-09T00:05:05.057274+0000","last_peered":"2024-03-09T00:05:05.057274+0000","last_clean":"2024-03-09T00:05:05.057274+0000","last_became_active":"2024-03-09T00:04:13.333835+0000","last_became_peered":"2024-03-09T00:04:13.333835+0000","last_unstale":"2024-03-09T00:05:05.057274+0000","last_undegraded":"2024-03-09T00:05:05.057274+0000","last_fullsized":"2024-03-09T00:05:05.057274+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:19:49.203289+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136870+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:05:05.136870+0000","last_peered":"2024-03-09T00:05:05.136870+0000","last_clean":"2024-03-09T00:05:05.136870+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:05:05.136870+0000","last_undegraded":"2024-03-09T00:05:05.136870+0000","last_fullsized":"2024-03-09T00:05:05.136870+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138319+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:05:05.138319+0000","last_peered":"2024-03-09T00:05:05.138319+0000","last_clean":"2024-03-09T00:05:05.138319+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:05:05.138319+0000","last_undegraded":"2024-03-09T00:05:05.138319+0000","last_fullsized":"2024-03-09T00:05:05.138319+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054365+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:05:05.054365+0000","last_peered":"2024-03-09T00:05:05.054365+0000","last_clean":"2024-03-09T00:05:05.054365+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:05:05.054365+0000","last_undegraded":"2024-03-09T00:05:05.054365+0000","last_fullsized":"2024-03-09T00:05:05.054365+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057072+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:05:05.057072+0000","last_peered":"2024-03-09T00:05:05.057072+0000","last_clean":"2024-03-09T00:05:05.057072+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:05:05.057072+0000","last_undegraded":"2024-03-09T00:05:05.057072+0000","last_fullsized":"2024-03-09T00:05:05.057072+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":12,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058517+0000","last_change":"2024-03-09T00:05:05.058517+0000","last_active":"2024-03-09T00:05:05.058517+0000","last_peered":"2024-03-09T00:05:05.058517+0000","last_clean":"2024-03-09T00:05:05.058517+0000","last_became_active":"2024-03-09T00:05:05.057877+0000","last_became_peered":"2024-03-09T00:05:05.057877+0000","last_unstale":"2024-03-09T00:05:05.058517+0000","last_undegraded":"2024-03-09T00:05:05.058517+0000","last_fullsized":"2024-03-09T00:05:05.058517+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:42:49.424790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"89'30","reported_seq":216,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:54.117874+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:05:54.117874+0000","last_peered":"2024-03-09T00:05:54.117874+0000","last_clean":"2024-03-09T00:05:54.117874+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:05:54.117874+0000","last_undegraded":"2024-03-09T00:05:54.117874+0000","last_fullsized":"2024-03-09T00:05:54.117874+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":30,"log_dups_size":0,"ondisk_log_size":30,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":63356,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":159,"num_read_kb":141,"num_write":30,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050774+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:05:05.050774+0000","last_peered":"2024-03-09T00:05:05.050774+0000","last_clean":"2024-03-09T00:05:05.050774+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:05:05.050774+0000","last_undegraded":"2024-03-09T00:05:05.050774+0000","last_fullsized":"2024-03-09T00:05:05.050774+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054832+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:05:05.054832+0000","last_peered":"2024-03-09T00:05:05.054832+0000","last_clean":"2024-03-09T00:05:05.054832+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:05:05.054832+0000","last_undegraded":"2024-03-09T00:05:05.054832+0000","last_fullsized":"2024-03-09T00:05:05.054832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053332+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:05:05.053332+0000","last_peered":"2024-03-09T00:05:05.053332+0000","last_clean":"2024-03-09T00:05:05.053332+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:05:05.053332+0000","last_undegraded":"2024-03-09T00:05:05.053332+0000","last_fullsized":"2024-03-09T00:05:05.053332+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052233+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:05:05.052233+0000","last_peered":"2024-03-09T00:05:05.052233+0000","last_clean":"2024-03-09T00:05:05.052233+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:05:05.052233+0000","last_undegraded":"2024-03-09T00:05:05.052233+0000","last_fullsized":"2024-03-09T00:05:05.052233+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054963+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:05:05.054963+0000","last_peered":"2024-03-09T00:05:05.054963+0000","last_clean":"2024-03-09T00:05:05.054963+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:05:05.054963+0000","last_undegraded":"2024-03-09T00:05:05.054963+0000","last_fullsized":"2024-03-09T00:05:05.054963+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053976+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:05:05.053976+0000","last_peered":"2024-03-09T00:05:05.053976+0000","last_clean":"2024-03-09T00:05:05.053976+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:05:05.053976+0000","last_undegraded":"2024-03-09T00:05:05.053976+0000","last_fullsized":"2024-03-09T00:05:05.053976+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.059283+0000","last_change":"2024-03-09T00:05:05.059283+0000","last_active":"2024-03-09T00:05:05.059283+0000","last_peered":"2024-03-09T00:05:05.059283+0000","last_clean":"2024-03-09T00:05:05.059283+0000","last_became_active":"2024-03-09T00:05:05.058822+0000","last_became_peered":"2024-03-09T00:05:05.058822+0000","last_unstale":"2024-03-09T00:05:05.059283+0000","last_undegraded":"2024-03-09T00:05:05.059283+0000","last_fullsized":"2024-03-09T00:05:05.059283+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:13:02.253810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051710+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:05:05.051710+0000","last_peered":"2024-03-09T00:05:05.051710+0000","last_clean":"2024-03-09T00:05:05.051710+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:05:05.051710+0000","last_undegraded":"2024-03-09T00:05:05.051710+0000","last_fullsized":"2024-03-09T00:05:05.051710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.137995+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:05:05.137995+0000","last_peered":"2024-03-09T00:05:05.137995+0000","last_clean":"2024-03-09T00:05:05.137995+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:05:05.137995+0000","last_undegraded":"2024-03-09T00:05:05.137995+0000","last_fullsized":"2024-03-09T00:05:05.137995+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056740+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:05:05.056740+0000","last_peered":"2024-03-09T00:05:05.056740+0000","last_clean":"2024-03-09T00:05:05.056740+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:05:05.056740+0000","last_undegraded":"2024-03-09T00:05:05.056740+0000","last_fullsized":"2024-03-09T00:05:05.056740+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052918+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:05:05.052918+0000","last_peered":"2024-03-09T00:05:05.052918+0000","last_clean":"2024-03-09T00:05:05.052918+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:05:05.052918+0000","last_undegraded":"2024-03-09T00:05:05.052918+0000","last_fullsized":"2024-03-09T00:05:05.052918+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":35,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049721+0000","last_change":"2024-03-09T00:04:13.330355+0000","last_active":"2024-03-09T00:05:05.049721+0000","last_peered":"2024-03-09T00:05:05.049721+0000","last_clean":"2024-03-09T00:05:05.049721+0000","last_became_active":"2024-03-09T00:04:13.330012+0000","last_became_peered":"2024-03-09T00:04:13.330012+0000","last_unstale":"2024-03-09T00:05:05.049721+0000","last_undegraded":"2024-03-09T00:05:05.049721+0000","last_fullsized":"2024-03-09T00:05:05.049721+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:36:46.699807+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053962+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:05:05.053962+0000","last_peered":"2024-03-09T00:05:05.053962+0000","last_clean":"2024-03-09T00:05:05.053962+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:05:05.053962+0000","last_undegraded":"2024-03-09T00:05:05.053962+0000","last_fullsized":"2024-03-09T00:05:05.053962+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052971+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:05:05.052971+0000","last_peered":"2024-03-09T00:05:05.052971+0000","last_clean":"2024-03-09T00:05:05.052971+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:05:05.052971+0000","last_undegraded":"2024-03-09T00:05:05.052971+0000","last_fullsized":"2024-03-09T00:05:05.052971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":29,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051325+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:05:05.051325+0000","last_peered":"2024-03-09T00:05:05.051325+0000","last_clean":"2024-03-09T00:05:05.051325+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:05:05.051325+0000","last_undegraded":"2024-03-09T00:05:05.051325+0000","last_fullsized":"2024-03-09T00:05:05.051325+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138598+0000","last_change":"2024-03-09T00:05:05.138598+0000","last_active":"2024-03-09T00:05:05.138598+0000","last_peered":"2024-03-09T00:05:05.138598+0000","last_clean":"2024-03-09T00:05:05.138598+0000","last_became_active":"2024-03-09T00:05:05.138166+0000","last_became_peered":"2024-03-09T00:05:05.138166+0000","last_unstale":"2024-03-09T00:05:05.138598+0000","last_undegraded":"2024-03-09T00:05:05.138598+0000","last_fullsized":"2024-03-09T00:05:05.138598+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:59:42.864722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050053+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:05:05.050053+0000","last_peered":"2024-03-09T00:05:05.050053+0000","last_clean":"2024-03-09T00:05:05.050053+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:05:05.050053+0000","last_undegraded":"2024-03-09T00:05:05.050053+0000","last_fullsized":"2024-03-09T00:05:05.050053+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051497+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:05:05.051497+0000","last_peered":"2024-03-09T00:05:05.051497+0000","last_clean":"2024-03-09T00:05:05.051497+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:05:05.051497+0000","last_undegraded":"2024-03-09T00:05:05.051497+0000","last_fullsized":"2024-03-09T00:05:05.051497+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053406+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:05:05.053406+0000","last_peered":"2024-03-09T00:05:05.053406+0000","last_clean":"2024-03-09T00:05:05.053406+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:05:05.053406+0000","last_undegraded":"2024-03-09T00:05:05.053406+0000","last_fullsized":"2024-03-09T00:05:05.053406+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052732+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:05:05.052732+0000","last_peered":"2024-03-09T00:05:05.052732+0000","last_clean":"2024-03-09T00:05:05.052732+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:05:05.052732+0000","last_undegraded":"2024-03-09T00:05:05.052732+0000","last_fullsized":"2024-03-09T00:05:05.052732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057196+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:05:05.057196+0000","last_peered":"2024-03-09T00:05:05.057196+0000","last_clean":"2024-03-09T00:05:05.057196+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:05:05.057196+0000","last_undegraded":"2024-03-09T00:05:05.057196+0000","last_fullsized":"2024-03-09T00:05:05.057196+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053098+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:05:05.053098+0000","last_peered":"2024-03-09T00:05:05.053098+0000","last_clean":"2024-03-09T00:05:05.053098+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:05:05.053098+0000","last_undegraded":"2024-03-09T00:05:05.053098+0000","last_fullsized":"2024-03-09T00:05:05.053098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138164+0000","last_change":"2024-03-09T00:05:05.138164+0000","last_active":"2024-03-09T00:05:05.138164+0000","last_peered":"2024-03-09T00:05:05.138164+0000","last_clean":"2024-03-09T00:05:05.138164+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:05:05.138164+0000","last_undegraded":"2024-03-09T00:05:05.138164+0000","last_fullsized":"2024-03-09T00:05:05.138164+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:12:44.310276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053931+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:05:05.053931+0000","last_peered":"2024-03-09T00:05:05.053931+0000","last_clean":"2024-03-09T00:05:05.053931+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:05:05.053931+0000","last_undegraded":"2024-03-09T00:05:05.053931+0000","last_fullsized":"2024-03-09T00:05:05.053931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052127+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:05:05.052127+0000","last_peered":"2024-03-09T00:05:05.052127+0000","last_clean":"2024-03-09T00:05:05.052127+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:05:05.052127+0000","last_undegraded":"2024-03-09T00:05:05.052127+0000","last_fullsized":"2024-03-09T00:05:05.052127+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056336+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:05:05.056336+0000","last_peered":"2024-03-09T00:05:05.056336+0000","last_clean":"2024-03-09T00:05:05.056336+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:05:05.056336+0000","last_undegraded":"2024-03-09T00:05:05.056336+0000","last_fullsized":"2024-03-09T00:05:05.056336+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":11,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.058661+0000","last_change":"2024-03-09T00:05:05.058253+0000","last_active":"2024-03-09T00:05:05.058661+0000","last_peered":"2024-03-09T00:05:05.058661+0000","last_clean":"2024-03-09T00:05:05.058661+0000","last_became_active":"2024-03-09T00:05:05.057864+0000","last_became_peered":"2024-03-09T00:05:05.057864+0000","last_unstale":"2024-03-09T00:05:05.058661+0000","last_undegraded":"2024-03-09T00:05:05.058661+0000","last_fullsized":"2024-03-09T00:05:05.058661+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:33:14.993355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138457+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:05:05.138457+0000","last_peered":"2024-03-09T00:05:05.138457+0000","last_clean":"2024-03-09T00:05:05.138457+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:05:05.138457+0000","last_undegraded":"2024-03-09T00:05:05.138457+0000","last_fullsized":"2024-03-09T00:05:05.138457+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":360,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056546+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:05:05.056546+0000","last_peered":"2024-03-09T00:05:05.056546+0000","last_clean":"2024-03-09T00:05:05.056546+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:05:05.056546+0000","last_undegraded":"2024-03-09T00:05:05.056546+0000","last_fullsized":"2024-03-09T00:05:05.056546+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053713+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:05:05.053713+0000","last_peered":"2024-03-09T00:05:05.053713+0000","last_clean":"2024-03-09T00:05:05.053713+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:05:05.053713+0000","last_undegraded":"2024-03-09T00:05:05.053713+0000","last_fullsized":"2024-03-09T00:05:05.053713+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052102+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:05:05.052102+0000","last_peered":"2024-03-09T00:05:05.052102+0000","last_clean":"2024-03-09T00:05:05.052102+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:05:05.052102+0000","last_undegraded":"2024-03-09T00:05:05.052102+0000","last_fullsized":"2024-03-09T00:05:05.052102+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056615+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:05:05.056615+0000","last_peered":"2024-03-09T00:05:05.056615+0000","last_clean":"2024-03-09T00:05:05.056615+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:05:05.056615+0000","last_undegraded":"2024-03-09T00:05:05.056615+0000","last_fullsized":"2024-03-09T00:05:05.056615+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054833+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:05:05.054833+0000","last_peered":"2024-03-09T00:05:05.054833+0000","last_clean":"2024-03-09T00:05:05.054833+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:05:05.054833+0000","last_undegraded":"2024-03-09T00:05:05.054833+0000","last_fullsized":"2024-03-09T00:05:05.054833+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050878+0000","last_change":"2024-03-09T00:04:13.331922+0000","last_active":"2024-03-09T00:05:05.050878+0000","last_peered":"2024-03-09T00:05:05.050878+0000","last_clean":"2024-03-09T00:05:05.050878+0000","last_became_active":"2024-03-09T00:04:13.331610+0000","last_became_peered":"2024-03-09T00:04:13.331610+0000","last_unstale":"2024-03-09T00:05:05.050878+0000","last_undegraded":"2024-03-09T00:05:05.050878+0000","last_fullsized":"2024-03-09T00:05:05.050878+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:39:29.706383+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052861+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:05:05.052861+0000","last_peered":"2024-03-09T00:05:05.052861+0000","last_clean":"2024-03-09T00:05:05.052861+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:05:05.052861+0000","last_undegraded":"2024-03-09T00:05:05.052861+0000","last_fullsized":"2024-03-09T00:05:05.052861+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049169+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:05:05.049169+0000","last_peered":"2024-03-09T00:05:05.049169+0000","last_clean":"2024-03-09T00:05:05.049169+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:05:05.049169+0000","last_undegraded":"2024-03-09T00:05:05.049169+0000","last_fullsized":"2024-03-09T00:05:05.049169+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"89'6","reported_seq":33,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051655+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:05:05.051655+0000","last_peered":"2024-03-09T00:05:05.051655+0000","last_clean":"2024-03-09T00:05:05.051655+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:05:05.051655+0000","last_undegraded":"2024-03-09T00:05:05.051655+0000","last_fullsized":"2024-03-09T00:05:05.051655+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":11,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049979+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:05:05.049979+0000","last_peered":"2024-03-09T00:05:05.049979+0000","last_clean":"2024-03-09T00:05:05.049979+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:05:05.049979+0000","last_undegraded":"2024-03-09T00:05:05.049979+0000","last_fullsized":"2024-03-09T00:05:05.049979+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052356+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:05:05.052356+0000","last_peered":"2024-03-09T00:05:05.052356+0000","last_clean":"2024-03-09T00:05:05.052356+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:05:05.052356+0000","last_undegraded":"2024-03-09T00:05:05.052356+0000","last_fullsized":"2024-03-09T00:05:05.052356+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138751+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:05:05.138751+0000","last_peered":"2024-03-09T00:05:05.138751+0000","last_clean":"2024-03-09T00:05:05.138751+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:05:05.138751+0000","last_undegraded":"2024-03-09T00:05:05.138751+0000","last_fullsized":"2024-03-09T00:05:05.138751+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136810+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:05:05.136810+0000","last_peered":"2024-03-09T00:05:05.136810+0000","last_clean":"2024-03-09T00:05:05.136810+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:05:05.136810+0000","last_undegraded":"2024-03-09T00:05:05.136810+0000","last_fullsized":"2024-03-09T00:05:05.136810+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138501+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:05:05.138501+0000","last_peered":"2024-03-09T00:05:05.138501+0000","last_clean":"2024-03-09T00:05:05.138501+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:05:05.138501+0000","last_undegraded":"2024-03-09T00:05:05.138501+0000","last_fullsized":"2024-03-09T00:05:05.138501+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055772+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:05:05.055772+0000","last_peered":"2024-03-09T00:05:05.055772+0000","last_clean":"2024-03-09T00:05:05.055772+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:05:05.055772+0000","last_undegraded":"2024-03-09T00:05:05.055772+0000","last_fullsized":"2024-03-09T00:05:05.055772+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049781+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:05:05.049781+0000","last_peered":"2024-03-09T00:05:05.049781+0000","last_clean":"2024-03-09T00:05:05.049781+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:05:05.049781+0000","last_undegraded":"2024-03-09T00:05:05.049781+0000","last_fullsized":"2024-03-09T00:05:05.049781+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051777+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:05:05.051777+0000","last_peered":"2024-03-09T00:05:05.051777+0000","last_clean":"2024-03-09T00:05:05.051777+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:05:05.051777+0000","last_undegraded":"2024-03-09T00:05:05.051777+0000","last_fullsized":"2024-03-09T00:05:05.051777+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":26,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049358+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:05:05.049358+0000","last_peered":"2024-03-09T00:05:05.049358+0000","last_clean":"2024-03-09T00:05:05.049358+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:05:05.049358+0000","last_undegraded":"2024-03-09T00:05:05.049358+0000","last_fullsized":"2024-03-09T00:05:05.049358+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054636+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:05:05.054636+0000","last_peered":"2024-03-09T00:05:05.054636+0000","last_clean":"2024-03-09T00:05:05.054636+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:05:05.054636+0000","last_undegraded":"2024-03-09T00:05:05.054636+0000","last_fullsized":"2024-03-09T00:05:05.054636+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":19,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138261+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:05:05.138261+0000","last_peered":"2024-03-09T00:05:05.138261+0000","last_clean":"2024-03-09T00:05:05.138261+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:05:05.138261+0000","last_undegraded":"2024-03-09T00:05:05.138261+0000","last_fullsized":"2024-03-09T00:05:05.138261+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051370+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:05:05.051370+0000","last_peered":"2024-03-09T00:05:05.051370+0000","last_clean":"2024-03-09T00:05:05.051370+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:05:05.051370+0000","last_undegraded":"2024-03-09T00:05:05.051370+0000","last_fullsized":"2024-03-09T00:05:05.051370+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049312+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:05:05.049312+0000","last_peered":"2024-03-09T00:05:05.049312+0000","last_clean":"2024-03-09T00:05:05.049312+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:05:05.049312+0000","last_undegraded":"2024-03-09T00:05:05.049312+0000","last_fullsized":"2024-03-09T00:05:05.049312+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.138730+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:05:05.138730+0000","last_peered":"2024-03-09T00:05:05.138730+0000","last_clean":"2024-03-09T00:05:05.138730+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:05:05.138730+0000","last_undegraded":"2024-03-09T00:05:05.138730+0000","last_fullsized":"2024-03-09T00:05:05.138730+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":30,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050211+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:05:05.050211+0000","last_peered":"2024-03-09T00:05:05.050211+0000","last_clean":"2024-03-09T00:05:05.050211+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:05:05.050211+0000","last_undegraded":"2024-03-09T00:05:05.050211+0000","last_fullsized":"2024-03-09T00:05:05.050211+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057000+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:05:05.057000+0000","last_peered":"2024-03-09T00:05:05.057000+0000","last_clean":"2024-03-09T00:05:05.057000+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:05:05.057000+0000","last_undegraded":"2024-03-09T00:05:05.057000+0000","last_fullsized":"2024-03-09T00:05:05.057000+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055823+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:05:05.055823+0000","last_peered":"2024-03-09T00:05:05.055823+0000","last_clean":"2024-03-09T00:05:05.055823+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:05:05.055823+0000","last_undegraded":"2024-03-09T00:05:05.055823+0000","last_fullsized":"2024-03-09T00:05:05.055823+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":27,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053442+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:05:05.053442+0000","last_peered":"2024-03-09T00:05:05.053442+0000","last_clean":"2024-03-09T00:05:05.053442+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:05:05.053442+0000","last_undegraded":"2024-03-09T00:05:05.053442+0000","last_fullsized":"2024-03-09T00:05:05.053442+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056696+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:05:05.056696+0000","last_peered":"2024-03-09T00:05:05.056696+0000","last_clean":"2024-03-09T00:05:05.056696+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:05:05.056696+0000","last_undegraded":"2024-03-09T00:05:05.056696+0000","last_fullsized":"2024-03-09T00:05:05.056696+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136421+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:05:05.136421+0000","last_peered":"2024-03-09T00:05:05.136421+0000","last_clean":"2024-03-09T00:05:05.136421+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:05:05.136421+0000","last_undegraded":"2024-03-09T00:05:05.136421+0000","last_fullsized":"2024-03-09T00:05:05.136421+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054593+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:05:05.054593+0000","last_peered":"2024-03-09T00:05:05.054593+0000","last_clean":"2024-03-09T00:05:05.054593+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:05:05.054593+0000","last_undegraded":"2024-03-09T00:05:05.054593+0000","last_fullsized":"2024-03-09T00:05:05.054593+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057094+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:05:05.057094+0000","last_peered":"2024-03-09T00:05:05.057094+0000","last_clean":"2024-03-09T00:05:05.057094+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:05:05.057094+0000","last_undegraded":"2024-03-09T00:05:05.057094+0000","last_fullsized":"2024-03-09T00:05:05.057094+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":18,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053282+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:05:05.053282+0000","last_peered":"2024-03-09T00:05:05.053282+0000","last_clean":"2024-03-09T00:05:05.053282+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:05:05.053282+0000","last_undegraded":"2024-03-09T00:05:05.053282+0000","last_fullsized":"2024-03-09T00:05:05.053282+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":28,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053043+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:05:05.053043+0000","last_peered":"2024-03-09T00:05:05.053043+0000","last_clean":"2024-03-09T00:05:05.053043+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:05:05.053043+0000","last_undegraded":"2024-03-09T00:05:05.053043+0000","last_fullsized":"2024-03-09T00:05:05.053043+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.057584+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:05:05.057584+0000","last_peered":"2024-03-09T00:05:05.057584+0000","last_clean":"2024-03-09T00:05:05.057584+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:05:05.057584+0000","last_undegraded":"2024-03-09T00:05:05.057584+0000","last_fullsized":"2024-03-09T00:05:05.057584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.054743+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:05:05.054743+0000","last_peered":"2024-03-09T00:05:05.054743+0000","last_clean":"2024-03-09T00:05:05.054743+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:05:05.054743+0000","last_undegraded":"2024-03-09T00:05:05.054743+0000","last_fullsized":"2024-03-09T00:05:05.054743+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052584+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:05:05.052584+0000","last_peered":"2024-03-09T00:05:05.052584+0000","last_clean":"2024-03-09T00:05:05.052584+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:05:05.052584+0000","last_undegraded":"2024-03-09T00:05:05.052584+0000","last_fullsized":"2024-03-09T00:05:05.052584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051228+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:05:05.051228+0000","last_peered":"2024-03-09T00:05:05.051228+0000","last_clean":"2024-03-09T00:05:05.051228+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:05:05.051228+0000","last_undegraded":"2024-03-09T00:05:05.051228+0000","last_fullsized":"2024-03-09T00:05:05.051228+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.136717+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:05:05.136717+0000","last_peered":"2024-03-09T00:05:05.136717+0000","last_clean":"2024-03-09T00:05:05.136717+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:05:05.136717+0000","last_undegraded":"2024-03-09T00:05:05.136717+0000","last_fullsized":"2024-03-09T00:05:05.136717+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052834+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:05:05.052834+0000","last_peered":"2024-03-09T00:05:05.052834+0000","last_clean":"2024-03-09T00:05:05.052834+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:05:05.052834+0000","last_undegraded":"2024-03-09T00:05:05.052834+0000","last_fullsized":"2024-03-09T00:05:05.052834+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055931+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:05:05.055931+0000","last_peered":"2024-03-09T00:05:05.055931+0000","last_clean":"2024-03-09T00:05:05.055931+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:05:05.055931+0000","last_undegraded":"2024-03-09T00:05:05.055931+0000","last_fullsized":"2024-03-09T00:05:05.055931+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.051815+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:05:05.051815+0000","last_peered":"2024-03-09T00:05:05.051815+0000","last_clean":"2024-03-09T00:05:05.051815+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:05:05.051815+0000","last_undegraded":"2024-03-09T00:05:05.051815+0000","last_fullsized":"2024-03-09T00:05:05.051815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.050101+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:05:05.050101+0000","last_peered":"2024-03-09T00:05:05.050101+0000","last_clean":"2024-03-09T00:05:05.050101+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:05:05.050101+0000","last_undegraded":"2024-03-09T00:05:05.050101+0000","last_fullsized":"2024-03-09T00:05:05.050101+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.053808+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:05:05.053808+0000","last_peered":"2024-03-09T00:05:05.053808+0000","last_clean":"2024-03-09T00:05:05.053808+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:05:05.053808+0000","last_undegraded":"2024-03-09T00:05:05.053808+0000","last_fullsized":"2024-03-09T00:05:05.053808+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.049386+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:05:05.049386+0000","last_peered":"2024-03-09T00:05:05.049386+0000","last_clean":"2024-03-09T00:05:05.049386+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:05:05.049386+0000","last_undegraded":"2024-03-09T00:05:05.049386+0000","last_fullsized":"2024-03-09T00:05:05.049386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":25,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.056305+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:05:05.056305+0000","last_peered":"2024-03-09T00:05:05.056305+0000","last_clean":"2024-03-09T00:05:05.056305+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:05:05.056305+0000","last_undegraded":"2024-03-09T00:05:05.056305+0000","last_fullsized":"2024-03-09T00:05:05.056305+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.052606+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:05:05.052606+0000","last_peered":"2024-03-09T00:05:05.052606+0000","last_clean":"2024-03-09T00:05:05.052606+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:05:05.052606+0000","last_undegraded":"2024-03-09T00:05:05.052606+0000","last_fullsized":"2024-03-09T00:05:05.052606+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":24,"reported_epoch":91,"state":"active+clean","last_fresh":"2024-03-09T00:05:05.055098+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:05:05.055098+0000","last_peered":"2024-03-09T00:05:05.055098+0000","last_clean":"2024-03-09T00:05:05.055098+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:05:05.055098+0000","last_undegraded":"2024-03-09T00:05:05.055098+0000","last_fullsized":"2024-03-09T00:05:05.055098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":64154,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":165,"num_read_kb":147,"num_write":79,"num_write_kb":90,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":282624,"data_stored":192462,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":68,"ondisk_log_size":68,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383715,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1019904,"data_stored":500249,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645352,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32840,"kb_used_data":1656,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712312,"statfs":{"total":95995035648,"available":95961407488,"internally_reserved":0,"allocated":1695744,"data_stored":1150071,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808877,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36348,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708804,"statfs":{"total":95995035648,"available":95957815296,"internally_reserved":0,"allocated":1028096,"data_stored":504927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070515,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1004,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1028096,"data_stored":500853,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332151,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32768,"kb_used_data":1584,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712384,"statfs":{"total":95995035648,"available":95961481216,"internally_reserved":0,"allocated":1622016,"data_stored":1095295,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":31914442},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495677,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":992,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1015808,"data_stored":496153,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574934,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32244,"kb_used_data":1064,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712908,"statfs":{"total":95995035648,"available":95962017792,"internally_reserved":0,"allocated":1089536,"data_stored":559509,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738458,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1019904,"data_stored":496243,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411409,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28092,"kb_used_data":1068,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717060,"statfs":{"total":95995035648,"available":95966269440,"internally_reserved":0,"allocated":1093632,"data_stored":559543,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182475,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28020,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717132,"statfs":{"total":95995035648,"available":95966343168,"internally_reserved":0,"allocated":1024000,"data_stored":496257,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920839,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36340,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708812,"statfs":{"total":95995035648,"available":95957823488,"internally_reserved":0,"allocated":1024000,"data_stored":500271,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757313,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36932,"kb_used_data":1588,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708220,"statfs":{"total":95995035648,"available":95957217280,"internally_reserved":0,"allocated":1626112,"data_stored":1090799,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63356,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63390,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":63550,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:05:59.402 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-09T00:05:59.402 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-09T00:05:59.402 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-09T00:05:59.402 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph health --format=json 2024-03-09T00:05:59.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:05:59 smithi102 ceph-mon[36106]: pgmap v348: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:05:59.646 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:00 smithi179 ceph-mon[36637]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:00 smithi073 ceph-mon[32309]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:00.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:00 smithi102 ceph-mon[36106]: from='client.15126 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:01.272 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:01 smithi073 ceph-mon[32309]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:01.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:01 smithi179 ceph-mon[36637]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:01.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:01 smithi102 ceph-mon[36106]: pgmap v349: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:01.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/3444901972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c101e30 msgr2=0x7f7b9c1022b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/3444901972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1022b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7f7b880099c0 tx=0x7f7b88009cd0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/3444901972 shutdown_connections 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/3444901972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c10ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/3444901972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 0x7f7b9c1081f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/3444901972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1022b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/3444901972 >> 172.21.15.73:0/3444901972 conn(0x7f7b9c078f10 msgr2=0x7f7b9c079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/3444901972 shutdown_connections 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.524+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/3444901972 wait complete. 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 Processor -- start 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 -- start start 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1985a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:01.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 0x7f7b9c19ce20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c112090 con 0x7f7b9c1027f0 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c111f10 con 0x7f7b9c101e30 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba2f0a700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b9c112210 con 0x7f7b9c107e10 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60974/0 (socket says 172.21.15.73:60974) 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 -- 172.21.15.73:0/2992905397 learned_addr learned my addr 172.21.15.73:0/2992905397 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba0ca6700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1985a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7ba14a7700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 0x7f7b9c19ce20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 -- 172.21.15.73:0/2992905397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 msgr2=0x7f7b9c19ce20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 0x7f7b9c19ce20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 -- 172.21.15.73:0/2992905397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c101e30 msgr2=0x7f7b9c1985a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1985a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 -- 172.21.15.73:0/2992905397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b9c19d560 con 0x7f7b9c1027f0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.525+0000 7f7b9bfff700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006010 tx=0x7f7b88009f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.526+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88027070 con 0x7f7b9c1027f0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.526+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b9c19d7f0 con 0x7f7b9c1027f0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.526+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b88004050 con 0x7f7b9c1027f0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.526+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b88022e10 con 0x7f7b9c1027f0 2024-03-09T00:06:01.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.526+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b9c1a50d0 con 0x7f7b9c1027f0 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.527+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b9c068aa0 con 0x7f7b9c1027f0 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.528+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f7b8802a070 con 0x7f7b9c1027f0 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.529+0000 7f7b99ffb700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 0x7f7b8c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.529+0000 7f7ba0ca6700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 0x7f7b8c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.529+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f7b880cfea0 con 0x7f7b9c1027f0 2024-03-09T00:06:01.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.529+0000 7f7ba0ca6700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 0x7f7b8c0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b90005950 tx=0x7f7b900058e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:01.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.533+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f7b88013070 con 0x7f7b9c1027f0 2024-03-09T00:06:01.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.768+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7b9c1998b0 con 0x7f7b9c1027f0 2024-03-09T00:06:01.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.769+0000 7f7b99ffb700 1 -- 172.21.15.73:0/2992905397 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f7b8809cb10 con 0x7f7b9c1027f0 2024-03-09T00:06:01.764 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:01.764 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 msgr2=0x7f7b8c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 0x7f7b8c0a4de0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7f7b90005950 tx=0x7f7b900058e0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 msgr2=0x7f7b9c198ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7f7b88006010 tx=0x7f7b88009f80 comp rx=0 tx=0).stop 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 shutdown_connections 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f7b8c0a2920 0x7f7b8c0a4de0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7b9c107e10 0x7f7b9c19ce20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7b9c101e30 0x7f7b9c1985a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 --2- 172.21.15.73:0/2992905397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7b9c1027f0 0x7f7b9c198ae0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:01.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 >> 172.21.15.73:0/2992905397 conn(0x7f7b9c078f10 msgr2=0x7f7b9c100b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:01.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 shutdown_connections 2024-03-09T00:06:01.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:01.771+0000 7f7ba2f0a700 1 -- 172.21.15.73:0/2992905397 wait complete. 2024-03-09T00:06:02.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:02 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2992905397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-09T00:06:02.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:02 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2992905397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-09T00:06:02.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:02 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2992905397' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-09T00:06:02.555 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-09T00:06:02.555 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:02.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 -- 172.21.15.73:0/2129528712 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc107070 msgr2=0x7f60bc1094c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2129528712 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc107070 0x7f60bc1094c0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f60b4009ca0 tx=0x7f60b402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 -- 172.21.15.73:0/2129528712 shutdown_connections 2024-03-09T00:06:02.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2129528712 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc107070 0x7f60bc1094c0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2129528712 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2129528712 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc101e10 0x7f60bc104200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 -- 172.21.15.73:0/2129528712 >> 172.21.15.73:0/2129528712 conn(0x7f60bc0fb820 msgr2=0x7f60bc0fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 -- 172.21.15.73:0/2129528712 shutdown_connections 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.665+0000 7f60c26e9640 1 -- 172.21.15.73:0/2129528712 wait complete. 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 Processor -- start 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 -- start start 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 0x7f60bc19eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc10cfb0 con 0x7f60bc101e10 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc10ce30 con 0x7f60bc107070 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c26e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc10d130 con 0x7f60bc104740 2024-03-09T00:06:02.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60994/0 (socket says 172.21.15.73:60994) 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 -- 172.21.15.73:0/2095642566 learned_addr learned my addr 172.21.15.73:0/2095642566 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60c0c86640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 0x7f60bc19eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bb7fe640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 -- 172.21.15.73:0/2095642566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 msgr2=0x7f60bc197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 -- 172.21.15.73:0/2095642566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 msgr2=0x7f60bc19eb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.666+0000 7f60bbfff640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 0x7f60bc19eb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60bbfff640 1 -- 172.21.15.73:0/2095642566 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60bc19f200 con 0x7f60bc101e10 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60c0c86640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 0x7f60bc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60bb7fe640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60bbfff640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00d950 tx=0x7f60ac00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac017070 con 0x7f60bc101e10 2024-03-09T00:06:02.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60ac009c30 con 0x7f60bc101e10 2024-03-09T00:06:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc19f4f0 con 0x7f60bc101e10 2024-03-09T00:06:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60ac00f6b0 con 0x7f60bc101e10 2024-03-09T00:06:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.667+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc19fa40 con 0x7f60bc101e10 2024-03-09T00:06:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.668+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc0fd310 con 0x7f60bc101e10 2024-03-09T00:06:02.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.669+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f60ac0109d0 con 0x7f60bc101e10 2024-03-09T00:06:02.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.669+0000 7f60b97fa640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 0x7f60a40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.670+0000 7f60bb7fe640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 0x7f60a40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.670+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f60ac0cecb0 con 0x7f60bc101e10 2024-03-09T00:06:02.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.670+0000 7f60bb7fe640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 0x7f60a40a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f60bc198af0 tx=0x7f60b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:02.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.674+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f60ac009de0 con 0x7f60bc101e10 2024-03-09T00:06:02.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.847+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60bc061ef0 con 0x7f60bc101e10 2024-03-09T00:06:02.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.849+0000 7f60b97fa640 1 -- 172.21.15.73:0/2095642566 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24585 (secure 0 0 0) 0x7f60ac09b890 con 0x7f60bc101e10 2024-03-09T00:06:02.844 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:02.844 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":91,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:05:05.042269+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:02.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 msgr2=0x7f60a40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 0x7f60a40a4d50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f60bc198af0 tx=0x7f60b0005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:02.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 msgr2=0x7f60bc197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f60ac00d950 tx=0x7f60ac00de10 comp rx=0 tx=0).stop 2024-03-09T00:06:02.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 shutdown_connections 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f60a40a2890 0x7f60a40a4d50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60bc107070 0x7f60bc19eb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60bc104740 0x7f60bc197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 --2- 172.21.15.73:0/2095642566 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60bc101e10 0x7f60bc197650 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 >> 172.21.15.73:0/2095642566 conn(0x7f60bc0fb820 msgr2=0x7f60bc1027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 shutdown_connections 2024-03-09T00:06:02.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.851+0000 7f60c26e9640 1 -- 172.21.15.73:0/2095642566 wait complete. 2024-03-09T00:06:02.857 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:02.858 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:02.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 -- 172.21.15.73:0/1535838318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 msgr2=0x7f9198102980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 --2- 172.21.15.73:0/1535838318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f9198102980 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f918c009900 tx=0x7f918c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:02.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 -- 172.21.15.73:0/1535838318 shutdown_connections 2024-03-09T00:06:02.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 --2- 172.21.15.73:0/1535838318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 --2- 172.21.15.73:0/1535838318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f9198102980 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 --2- 172.21.15.73:0/1535838318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f9198101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 -- 172.21.15.73:0/1535838318 >> 172.21.15.73:0/1535838318 conn(0x7f91980fc880 msgr2=0x7f91980fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 -- 172.21.15.73:0/1535838318 shutdown_connections 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.969+0000 7f919e51b640 1 -- 172.21.15.73:0/1535838318 wait complete. 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 Processor -- start 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 -- start start 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f91980750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919810cfa0 con 0x7f9198102500 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919810ce20 con 0x7f9198101310 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919e51b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f919810d120 con 0x7f9198102ec0 2024-03-09T00:06:02.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32778/0 (socket says 172.21.15.73:32778) 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 -- 172.21.15.73:0/3291436490 learned_addr learned my addr 172.21.15.73:0/3291436490 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919cab8640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f9197fff640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f91980750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 -- 172.21.15.73:0/3291436490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 msgr2=0x7f9198073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 -- 172.21.15.73:0/3291436490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 msgr2=0x7f91980750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f91980750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f91977fe640 1 -- 172.21.15.73:0/3291436490 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91980744b0 con 0x7f9198102500 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f919cab8640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:02.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.970+0000 7f9197fff640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f91980750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f91977fe640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f918c009230 tx=0x7f918c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c041070 con 0x7f9198102500 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f918c0044d0 con 0x7f9198102500 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f918c034440 con 0x7f9198102500 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91981a1630 con 0x7f9198102500 2024-03-09T00:06:02.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.971+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91981a1b80 con 0x7f9198102500 2024-03-09T00:06:02.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.972+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91981085d0 con 0x7f9198102500 2024-03-09T00:06:02.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.973+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f918c044070 con 0x7f9198102500 2024-03-09T00:06:02.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.973+0000 7f91957fa640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 0x7f91800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:02.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.973+0000 7f9197fff640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 0x7f91800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:02.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.973+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f918c0f0b70 con 0x7f9198102500 2024-03-09T00:06:02.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.974+0000 7f9197fff640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 0x7f91800a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9198102360 tx=0x7f9188005db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:02.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:02.978+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f918c0bd860 con 0x7f9198102500 2024-03-09T00:06:03.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.151+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f919802ce70 con 0x7f9198102500 2024-03-09T00:06:03.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.151+0000 7f91957fa640 1 -- 172.21.15.73:0/3291436490 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f918c0bd160 con 0x7f9198102500 2024-03-09T00:06:03.146 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:03.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 msgr2=0x7f91800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 0x7f91800a4d50 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f9198102360 tx=0x7f9188005db0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 msgr2=0x7f91980736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f918c009230 tx=0x7f918c005d50 comp rx=0 tx=0).stop 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 shutdown_connections 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f91800a2890 0x7f91800a4d50 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9198102ec0 0x7f9198073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9198102500 0x7f91980736e0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 --2- 172.21.15.73:0/3291436490 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9198101310 0x7f91980750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 >> 172.21.15.73:0/3291436490 conn(0x7f91980fc880 msgr2=0x7f91981057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 shutdown_connections 2024-03-09T00:06:03.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.153+0000 7f919e51b640 1 -- 172.21.15.73:0/3291436490 wait complete. 2024-03-09T00:06:03.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:03 smithi073 ceph-mon[32309]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:03.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:03 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2095642566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:03.159 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- 172.21.15.73:0/1039093338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd78101300 msgr2=0x7fbd78101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/1039093338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd78101300 0x7fbd78101720 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fbd68009900 tx=0x7fbd6802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- 172.21.15.73:0/1039093338 shutdown_connections 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/1039093338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78102eb0 0x7fbd78109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/1039093338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd781024f0 0x7fbd78102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/1039093338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd78101300 0x7fbd78101720 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- 172.21.15.73:0/1039093338 >> 172.21.15.73:0/1039093338 conn(0x7fbd780fc890 msgr2=0x7fbd780fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- 172.21.15.73:0/1039093338 shutdown_connections 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- 172.21.15.73:0/1039093338 wait complete. 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 Processor -- start 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 -- start start 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78101300 0x7fbd781975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.270+0000 7fbd7eb29640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78102eb0 0x7fbd7819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7810cfe0 con 0x7fbd781024f0 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7810ce60 con 0x7fbd78101300 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd7810d160 con 0x7fbd78102eb0 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32802/0 (socket says 172.21.15.73:32802) 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 -- 172.21.15.73:0/2445004676 learned_addr learned my addr 172.21.15.73:0/2445004676 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7c8c5640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78101300 0x7fbd781975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7d0c6640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78102eb0 0x7fbd7819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 -- 172.21.15.73:0/2445004676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78102eb0 msgr2=0x7fbd7819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78102eb0 0x7fbd7819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 -- 172.21.15.73:0/2445004676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78101300 msgr2=0x7fbd781975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78101300 0x7fbd781975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 -- 172.21.15.73:0/2445004676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd7819f1d0 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd77fff640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00c920 tx=0x7fbd6c00cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c007c70 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd6c007e10 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd7819f4c0 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd6c018890 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.271+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd7819fa60 con 0x7fbd781024f0 2024-03-09T00:06:03.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.272+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd781085c0 con 0x7fbd781024f0 2024-03-09T00:06:03.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.273+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fbd6c018a70 con 0x7fbd781024f0 2024-03-09T00:06:03.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.274+0000 7fbd75ffb640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 0x7fbd600a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.274+0000 7fbd7c8c5640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 0x7fbd600a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.274+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fbd6c0cef50 con 0x7fbd781024f0 2024-03-09T00:06:03.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.274+0000 7fbd7c8c5640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 0x7fbd600a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd6802c170 tx=0x7fbd6802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.278+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbd6c014030 con 0x7fbd781024f0 2024-03-09T00:06:03.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:03 smithi179 ceph-mon[36637]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:03.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:03 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2095642566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:03.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.453+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fbd7802d020 con 0x7fbd781024f0 2024-03-09T00:06:03.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.453+0000 7fbd75ffb640 1 -- 172.21.15.73:0/2445004676 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+11 (secure 0 0 0) 0x7fbd6c09bb30 con 0x7fbd781024f0 2024-03-09T00:06:03.448 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:03.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 msgr2=0x7fbd600a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 0x7fbd600a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fbd6802c170 tx=0x7fbd6802df90 comp rx=0 tx=0).stop 2024-03-09T00:06:03.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 msgr2=0x7fbd78197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fbd6c00c920 tx=0x7fbd6c00cde0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 shutdown_connections 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbd600a2910 0x7fbd600a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd78102eb0 0x7fbd7819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd781024f0 0x7fbd78197af0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 --2- 172.21.15.73:0/2445004676 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd78101300 0x7fbd781975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 >> 172.21.15.73:0/2445004676 conn(0x7fbd780fc890 msgr2=0x7fbd78105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.455+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 shutdown_connections 2024-03-09T00:06:03.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.456+0000 7fbd7eb29640 1 -- 172.21.15.73:0/2445004676 wait complete. 2024-03-09T00:06:03.461 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:03.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:03 smithi102 ceph-mon[36106]: pgmap v350: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:03.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:03 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2095642566' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:03.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.570+0000 7f0c7f553640 1 -- 172.21.15.73:0/2248022226 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c781001f0 msgr2=0x7f0c78100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.570+0000 7f0c7f553640 1 --2- 172.21.15.73:0/2248022226 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c781001f0 0x7f0c78100610 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7f0c70009ca0 tx=0x7f0c7002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.570+0000 7f0c7f553640 1 -- 172.21.15.73:0/2248022226 shutdown_connections 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.570+0000 7f0c7f553640 1 --2- 172.21.15.73:0/2248022226 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c78101da0 0x7f0c78108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.570+0000 7f0c7f553640 1 --2- 172.21.15.73:0/2248022226 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781013e0 0x7f0c78101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 --2- 172.21.15.73:0/2248022226 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c781001f0 0x7f0c78100610 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- 172.21.15.73:0/2248022226 >> 172.21.15.73:0/2248022226 conn(0x7f0c780fb780 msgr2=0x7f0c780fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- 172.21.15.73:0/2248022226 shutdown_connections 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- 172.21.15.73:0/2248022226 wait complete. 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 Processor -- start 2024-03-09T00:06:03.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- start start 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781001f0 0x7f0c78069020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c781013e0 0x7f0c7819c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810be80 con 0x7f0c78101da0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810bd00 con 0x7f0c781001f0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7f553640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c7810c000 con 0x7f0c781013e0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7daf0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7daf0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32820/0 (socket says 172.21.15.73:32820) 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.571+0000 7f0c7daf0640 1 -- 172.21.15.73:0/239465557 learned_addr learned my addr 172.21.15.73:0/239465557 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7d2ef640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781001f0 0x7f0c78069020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7caee640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c781013e0 0x7f0c7819c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 -- 172.21.15.73:0/239465557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c781013e0 msgr2=0x7f0c7819c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c781013e0 0x7f0c7819c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 -- 172.21.15.73:0/239465557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781001f0 msgr2=0x7f0c78069020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781001f0 0x7f0c78069020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 -- 172.21.15.73:0/239465557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7819f2b0 con 0x7f0c78101da0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7daf0640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c970 tx=0x7f0c7400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c74007bb0 con 0x7f0c78101da0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c74007d50 con 0x7f0c78101da0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c7819f5a0 con 0x7f0c78101da0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c7400f450 con 0x7f0c78101da0 2024-03-09T00:06:03.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.572+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c7819faf0 con 0x7f0c78101da0 2024-03-09T00:06:03.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.573+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c78107530 con 0x7f0c78101da0 2024-03-09T00:06:03.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.574+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f0c74004f00 con 0x7f0c78101da0 2024-03-09T00:06:03.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.575+0000 7f0c6e7fc640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 0x7f0c640a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.575+0000 7f0c7d2ef640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 0x7f0c640a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.575+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f0c740cf090 con 0x7f0c78101da0 2024-03-09T00:06:03.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.575+0000 7f0c7d2ef640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 0x7f0c640a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c7002c700 tx=0x7f0c70009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.578+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0c74014030 con 0x7f0c78101da0 2024-03-09T00:06:03.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.754+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0c7802ce70 con 0x7f0c78101da0 2024-03-09T00:06:03.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.755+0000 7f0c6e7fc640 1 -- 172.21.15.73:0/239465557 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+11 (secure 0 0 0) 0x7f0c7409bc70 con 0x7f0c78101da0 2024-03-09T00:06:03.750 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 msgr2=0x7f0c640a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 0x7f0c640a4d50 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0c7002c700 tx=0x7f0c70009360 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 msgr2=0x7f0c7819e9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7f0c7400c970 tx=0x7f0c7400ce30 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 shutdown_connections 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0c640a2890 0x7f0c640a4d50 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c78101da0 0x7f0c7819e9f0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c781013e0 0x7f0c7819c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 --2- 172.21.15.73:0/239465557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c781001f0 0x7f0c78069020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 >> 172.21.15.73:0/239465557 conn(0x7f0c780fb780 msgr2=0x7f0c78104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 shutdown_connections 2024-03-09T00:06:03.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.757+0000 7f0c7f553640 1 -- 172.21.15.73:0/239465557 wait complete. 2024-03-09T00:06:03.763 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:03.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.872+0000 7fecaa297640 1 -- 172.21.15.73:0/1574646808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41001f0 msgr2=0x7feca4100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.872+0000 7fecaa297640 1 --2- 172.21.15.73:0/1574646808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41001f0 0x7feca4100610 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fec94009900 tx=0x7fec9402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- 172.21.15.73:0/1574646808 shutdown_connections 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- 172.21.15.73:0/1574646808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca4101da0 0x7feca4108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- 172.21.15.73:0/1574646808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca41013e0 0x7feca4101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- 172.21.15.73:0/1574646808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41001f0 0x7feca4100610 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- 172.21.15.73:0/1574646808 >> 172.21.15.73:0/1574646808 conn(0x7feca40fb780 msgr2=0x7feca40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- 172.21.15.73:0/1574646808 shutdown_connections 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- 172.21.15.73:0/1574646808 wait complete. 2024-03-09T00:06:03.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 Processor -- start 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- start start 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca41001f0 0x7feca4197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca4101da0 0x7feca419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca410be80 con 0x7feca41013e0 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca410bd00 con 0x7feca41001f0 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.873+0000 7fecaa297640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca410c000 con 0x7feca4101da0 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32832/0 (socket says 172.21.15.73:32832) 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 -- 172.21.15.73:0/4084025838 learned_addr learned my addr 172.21.15.73:0/4084025838 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:03.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca8834640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca4101da0 0x7feca419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca3fff640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca41001f0 0x7feca4197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 -- 172.21.15.73:0/4084025838 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca4101da0 msgr2=0x7feca419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca4101da0 0x7feca419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 -- 172.21.15.73:0/4084025838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca41001f0 msgr2=0x7feca4197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca41001f0 0x7feca4197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 -- 172.21.15.73:0/4084025838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feca419f240 con 0x7feca41013e0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca37fe640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fec9800d770 tx=0x7fec9800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98017070 con 0x7feca41013e0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec98005500 con 0x7feca41013e0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca419f530 con 0x7feca41013e0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec98004e00 con 0x7feca41013e0 2024-03-09T00:06:03.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.874+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca419fa30 con 0x7feca41013e0 2024-03-09T00:06:03.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.875+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca419fc90 con 0x7feca41013e0 2024-03-09T00:06:03.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.876+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7fec98004fa0 con 0x7feca41013e0 2024-03-09T00:06:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.877+0000 7feca17fa640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 0x7fec8c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.877+0000 7feca3fff640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 0x7fec8c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.877+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7fec980cea40 con 0x7feca41013e0 2024-03-09T00:06:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.878+0000 7feca3fff640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 0x7fec8c0a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec9402c7b0 tx=0x7fec94005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:03.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:03.882+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fec9809b620 con 0x7feca41013e0 2024-03-09T00:06:04.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.086+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feca419fe60 con 0x7feca41013e0 2024-03-09T00:06:04.081 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.086+0000 7feca17fa640 1 -- 172.21.15.73:0/4084025838 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v91) v1 ==== 103+0+11 (secure 0 0 0) 0x7fec9809af20 con 0x7feca41013e0 2024-03-09T00:06:04.081 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:04.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 msgr2=0x7fec8c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 0x7fec8c0a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fec9402c7b0 tx=0x7fec94005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 msgr2=0x7feca4197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fec9800d770 tx=0x7fec9800dc30 comp rx=0 tx=0).stop 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 shutdown_connections 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fec8c0a2890 0x7fec8c0a4d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feca4101da0 0x7feca419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feca41013e0 0x7feca4197b60 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 --2- 172.21.15.73:0/4084025838 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feca41001f0 0x7feca4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 >> 172.21.15.73:0/4084025838 conn(0x7feca40fb780 msgr2=0x7feca4068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 shutdown_connections 2024-03-09T00:06:04.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.088+0000 7fecaa297640 1 -- 172.21.15.73:0/4084025838 wait complete. 2024-03-09T00:06:04.094 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 -- 172.21.15.73:0/3112166423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101230 msgr2=0x7f10a01016b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 --2- 172.21.15.73:0/3112166423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101230 0x7f10a01016b0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f1094009900 tx=0x7f109402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 -- 172.21.15.73:0/3112166423 shutdown_connections 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 --2- 172.21.15.73:0/3112166423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0101bf0 0x7f10a0108480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 --2- 172.21.15.73:0/3112166423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101230 0x7f10a01016b0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 --2- 172.21.15.73:0/3112166423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0100040 0x7f10a0100460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 -- 172.21.15.73:0/3112166423 >> 172.21.15.73:0/3112166423 conn(0x7f10a00fb5d0 msgr2=0x7f10a00fd9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 -- 172.21.15.73:0/3112166423 shutdown_connections 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 -- 172.21.15.73:0/3112166423 wait complete. 2024-03-09T00:06:04.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.202+0000 7f10a4da2640 1 Processor -- start 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 -- start start 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 0x7f10a01995f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0101230 0x7f10a0199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a010bcd0 con 0x7f10a0101bf0 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a010bb50 con 0x7f10a0100040 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f10a4da2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10a010be50 con 0x7f10a0101230 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32856/0 (socket says 172.21.15.73:32856) 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 -- 172.21.15.73:0/1520880569 learned_addr learned my addr 172.21.15.73:0/1520880569 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109effd640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0101230 0x7f10a0199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109f7fe640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 0x7f10a01995f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 -- 172.21.15.73:0/1520880569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0101230 msgr2=0x7f10a0199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0101230 0x7f10a0199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 -- 172.21.15.73:0/1520880569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 msgr2=0x7f10a01995f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 0x7f10a01995f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 -- 172.21.15.73:0/1520880569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10a01a1210 con 0x7f10a0101bf0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109f7fe640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 0x7f10a01995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.203+0000 7f109ffff640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f109800cce0 tx=0x7f109800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f109800f450 con 0x7f10a0101bf0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f109800fad0 con 0x7f10a0101bf0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10a01a1500 con 0x7f10a0101bf0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1098018c10 con 0x7f10a0101bf0 2024-03-09T00:06:04.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10a01a19d0 con 0x7f10a0101bf0 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.204+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10a01a1c90 con 0x7f10a0101bf0 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.206+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f109800f5f0 con 0x7f10a0101bf0 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.206+0000 7f109cff9640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 0x7f10880a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.206+0000 7f109f7fe640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 0x7f10880a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.206+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f10980cf710 con 0x7f10a0101bf0 2024-03-09T00:06:04.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.207+0000 7f109f7fe640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 0x7f10880a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f10a0101090 tx=0x7f1090005e70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.210+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1098014030 con 0x7f10a0101bf0 2024-03-09T00:06:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3291436490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2445004676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/239465557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3291436490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2445004676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/239465557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.422+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f10a01a1e60 con 0x7f10a0101bf0 2024-03-09T00:06:04.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.423+0000 7f109cff9640 1 -- 172.21.15.73:0/1520880569 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f109809c2f0 con 0x7f10a0101bf0 2024-03-09T00:06:04.418 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:04.418 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:04.419 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:04.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 msgr2=0x7f10880a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 0x7f10880a4d50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f10a0101090 tx=0x7f1090005e70 comp rx=0 tx=0).stop 2024-03-09T00:06:04.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 msgr2=0x7f10a01a0ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f109800cce0 tx=0x7f109800a620 comp rx=0 tx=0).stop 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 shutdown_connections 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f10880a2890 0x7f10880a4d50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f10a0101bf0 0x7f10a01a0ba0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f10a0101230 0x7f10a0199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 --2- 172.21.15.73:0/1520880569 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f10a0100040 0x7f10a01995f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 >> 172.21.15.73:0/1520880569 conn(0x7f10a00fb5d0 msgr2=0x7f10a00fd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 shutdown_connections 2024-03-09T00:06:04.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.425+0000 7f10a4da2640 1 -- 172.21.15.73:0/1520880569 wait complete. 2024-03-09T00:06:04.432 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3291436490' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2445004676' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/239465557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:04.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 -- 172.21.15.73:0/1283151082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c102ef0 msgr2=0x7f3b1c109780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 --2- 172.21.15.73:0/1283151082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c109780 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f3b14009ca0 tx=0x7f3b1402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 -- 172.21.15.73:0/1283151082 shutdown_connections 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 --2- 172.21.15.73:0/1283151082 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c109780 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 --2- 172.21.15.73:0/1283151082 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102530 0x7f3b1c1029b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 --2- 172.21.15.73:0/1283151082 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c101340 0x7f3b1c101760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 -- 172.21.15.73:0/1283151082 >> 172.21.15.73:0/1283151082 conn(0x7f3b1c0fc890 msgr2=0x7f3b1c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 -- 172.21.15.73:0/1283151082 shutdown_connections 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.539+0000 7f3b2317e640 1 -- 172.21.15.73:0/1283151082 wait complete. 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 Processor -- start 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 -- start start 2024-03-09T00:06:04.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 0x7f3b1c199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10cfd0 con 0x7f3b1c101340 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10ce50 con 0x7f3b1c102ef0 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2317e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b1c10d150 con 0x7f3b1c102530 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32874/0 (socket says 172.21.15.73:32874) 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 -- 172.21.15.73:0/3066299468 learned_addr learned my addr 172.21.15.73:0/3066299468 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b2171b640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b1bfff640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 0x7f3b1c199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 -- 172.21.15.73:0/3066299468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 msgr2=0x7f3b1c199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 0x7f3b1c199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 -- 172.21.15.73:0/3066299468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102ef0 msgr2=0x7f3b1c1a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.540+0000 7f3b20f1a640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c1a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b20f1a640 1 -- 172.21.15.73:0/3066299468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b1c1a1430 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b1bfff640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 0x7f3b1c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b20f1a640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c970 tx=0x7f3b0c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c004220 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b0c0043c0 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b1c1a1720 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b0c004a00 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.541+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b1c1a1c70 con 0x7f3b1c101340 2024-03-09T00:06:04.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.542+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b1c1a1f30 con 0x7f3b1c101340 2024-03-09T00:06:04.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.543+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f3b0c0079b0 con 0x7f3b1c101340 2024-03-09T00:06:04.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.544+0000 7f3b19ffb640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 0x7f3b040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.544+0000 7f3b1bfff640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 0x7f3b040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.544+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f3b0c0cecb0 con 0x7f3b1c101340 2024-03-09T00:06:04.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.544+0000 7f3b1bfff640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 0x7f3b040a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005fd0 tx=0x7f3b10005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.548+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3b0c014030 con 0x7f3b1c101340 2024-03-09T00:06:04.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.722+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3b1c1a2100 con 0x7f3b1c101340 2024-03-09T00:06:04.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.724+0000 7f3b19ffb640 1 -- 172.21.15.73:0/3066299468 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+24585 (secure 0 0 0) 0x7f3b0c09b890 con 0x7f3b1c101340 2024-03-09T00:06:04.719 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:04.719 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":91,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:05:05.042269+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 msgr2=0x7f3b040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 0x7f3b040a4dd0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f3b10005fd0 tx=0x7f3b10005ea0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 msgr2=0x7f3b1c199810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7f3b0c00c970 tx=0x7f3b0c00ce30 comp rx=0 tx=0).stop 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 shutdown_connections 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3b040a2910 0x7f3b040a4dd0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3b1c102ef0 0x7f3b1c1a0dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3b1c102530 0x7f3b1c199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 --2- 172.21.15.73:0/3066299468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3b1c101340 0x7f3b1c199810 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 >> 172.21.15.73:0/3066299468 conn(0x7f3b1c0fc890 msgr2=0x7f3b1c073520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 shutdown_connections 2024-03-09T00:06:04.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.726+0000 7f3b2317e640 1 -- 172.21.15.73:0/3066299468 wait complete. 2024-03-09T00:06:04.733 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:04.733 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:04.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 -- 172.21.15.73:0/1419689228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 msgr2=0x7f66e8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 --2- 172.21.15.73:0/1419689228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e8100610 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f66e4009ca0 tx=0x7f66e402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 -- 172.21.15.73:0/1419689228 shutdown_connections 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 --2- 172.21.15.73:0/1419689228 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e8101da0 0x7f66e8108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 --2- 172.21.15.73:0/1419689228 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e81013e0 0x7f66e8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 --2- 172.21.15.73:0/1419689228 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e8100610 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.842+0000 7f66eff45640 1 -- 172.21.15.73:0/1419689228 >> 172.21.15.73:0/1419689228 conn(0x7f66e80fb780 msgr2=0x7f66e80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- 172.21.15.73:0/1419689228 shutdown_connections 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- 172.21.15.73:0/1419689228 wait complete. 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 Processor -- start 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- start start 2024-03-09T00:06:04.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e81013e0 0x7f66e8197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 0x7f66e819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e810be80 con 0x7f66e81001f0 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e810bd00 con 0x7f66e81013e0 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66eff45640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66e810c000 con 0x7f66e8101da0 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66edce1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66edce1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32886/0 (socket says 172.21.15.73:32886) 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.843+0000 7f66edce1640 1 -- 172.21.15.73:0/1539506988 learned_addr learned my addr 172.21.15.73:0/1539506988 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:04.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 -- 172.21.15.73:0/1539506988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 msgr2=0x7f66e819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66ee4e2640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 0x7f66e819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 0x7f66e819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 -- 172.21.15.73:0/1539506988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e81013e0 msgr2=0x7f66e8197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e81013e0 0x7f66e8197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 -- 172.21.15.73:0/1539506988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e819f2d0 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66ee4e2640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 0x7f66e819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66edce1640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f66e40324c0 tx=0x7f66e4032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e4039040 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66e4044070 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66e819f560 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f66e40042a0 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.844+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66e819faa0 con 0x7f66e81001f0 2024-03-09T00:06:04.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.845+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66e819fdc0 con 0x7f66e81001f0 2024-03-09T00:06:04.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.846+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 17) v1 ==== 137341+0+0 (secure 0 0 0) 0x7f66e4003970 con 0x7f66e81001f0 2024-03-09T00:06:04.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.847+0000 7f66deffd640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 0x7f66d40a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:04.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.847+0000 7f66ed4e0640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 0x7f66d40a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:04.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.847+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 9092+0+0 (secure 0 0 0) 0x7f66e40f1dd0 con 0x7f66e81001f0 2024-03-09T00:06:04.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.847+0000 7f66ed4e0640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 0x7f66d40a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f66d8005fd0 tx=0x7f66d8005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:04.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:04.850+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f66e40beb50 con 0x7f66e81001f0 2024-03-09T00:06:05.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.023+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f66e819ff90 con 0x7f66e81001f0 2024-03-09T00:06:05.018 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.023+0000 7f66deffd640 1 -- 172.21.15.73:0/1539506988 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v91) v1 ==== 93+0+10 (secure 0 0 0) 0x7f66e40be450 con 0x7f66e81001f0 2024-03-09T00:06:05.018 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:05.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 msgr2=0x7f66d40a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 0x7f66d40a4dd0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f66d8005fd0 tx=0x7f66d8005850 comp rx=0 tx=0).stop 2024-03-09T00:06:05.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 msgr2=0x7f66e81976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f66e40324c0 tx=0x7f66e4032f20 comp rx=0 tx=0).stop 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 shutdown_connections 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f66d40a2910 0x7f66d40a4dd0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f66e8101da0 0x7f66e819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f66e81013e0 0x7f66e8197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 --2- 172.21.15.73:0/1539506988 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f66e81001f0 0x7f66e81976b0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 >> 172.21.15.73:0/1539506988 conn(0x7f66e80fb780 msgr2=0x7f66e8068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 shutdown_connections 2024-03-09T00:06:05.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.025+0000 7f66eff45640 1 -- 172.21.15.73:0/1539506988 wait complete. 2024-03-09T00:06:05.031 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 -- 172.21.15.73:0/1732106316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59881024f0 msgr2=0x7f5988102970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 --2- 172.21.15.73:0/1732106316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59881024f0 0x7f5988102970 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7f597c009900 tx=0x7f597c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 -- 172.21.15.73:0/1732106316 shutdown_connections 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 --2- 172.21.15.73:0/1732106316 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5988102eb0 0x7f5988109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 --2- 172.21.15.73:0/1732106316 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f59881024f0 0x7f5988102970 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 --2- 172.21.15.73:0/1732106316 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 0x7f5988101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 -- 172.21.15.73:0/1732106316 >> 172.21.15.73:0/1732106316 conn(0x7f59880fc890 msgr2=0x7f59880fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 -- 172.21.15.73:0/1732106316 shutdown_connections 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.147+0000 7f598e829640 1 -- 172.21.15.73:0/1732106316 wait complete. 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 Processor -- start 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 -- start start 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 0x7f5988197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59881024f0 0x7f5988197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598810cf90 con 0x7f5988102eb0 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598810ce10 con 0x7f5988101300 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598e829640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f598810d110 con 0x7f59881024f0 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598cdc6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598cdc6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32902/0 (socket says 172.21.15.73:32902) 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f598cdc6640 1 -- 172.21.15.73:0/3148256982 learned_addr learned my addr 172.21.15.73:0/3148256982 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:05.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f5987fff640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 0x7f5988197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.148+0000 7f59877fe640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59881024f0 0x7f5988197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 -- 172.21.15.73:0/3148256982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59881024f0 msgr2=0x7f5988197b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59881024f0 0x7f5988197b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 -- 172.21.15.73:0/3148256982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 msgr2=0x7f5988197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 0x7f5988197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 -- 172.21.15.73:0/3148256982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f598819f230 con 0x7f5988102eb0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598cdc6640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f598000eaa0 tx=0x7f598000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f598000cdb0 con 0x7f5988102eb0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f598819f520 con 0x7f5988102eb0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5980005020 con 0x7f5988102eb0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59800116b0 con 0x7f5988102eb0 2024-03-09T00:06:05.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.149+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f598819fa40 con 0x7f5988102eb0 2024-03-09T00:06:05.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.150+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5988108620 con 0x7f5988102eb0 2024-03-09T00:06:05.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.151+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5980005560 con 0x7f5988102eb0 2024-03-09T00:06:05.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.152+0000 7f59857fa640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 0x7f59700a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.152+0000 7f5987fff640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 0x7f59700a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.152+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f59800cf060 con 0x7f5988102eb0 2024-03-09T00:06:05.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.152+0000 7f5987fff640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 0x7f59700a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5988102350 tx=0x7f5978005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.156+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5980016030 con 0x7f5988102eb0 2024-03-09T00:06:05.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.332+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5988061ef0 con 0x7f5988102eb0 2024-03-09T00:06:05.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.332+0000 7f59857fa640 1 -- 172.21.15.73:0/3148256982 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v92) v1 ==== 104+0+11 (secure 0 0 0) 0x7f598009bd50 con 0x7f5988102eb0 2024-03-09T00:06:05.327 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:05.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.334+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 msgr2=0x7f59700a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.334+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 0x7f59700a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f5988102350 tx=0x7f5978005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.334+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 msgr2=0x7f598819ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.334+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f598000eaa0 tx=0x7f598000ef60 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 shutdown_connections 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59700a2800 0x7f59700a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5988102eb0 0x7f598819ebc0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f59881024f0 0x7f5988197b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 --2- 172.21.15.73:0/3148256982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5988101300 0x7f5988197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 >> 172.21.15.73:0/3148256982 conn(0x7f59880fc890 msgr2=0x7f59881057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 shutdown_connections 2024-03-09T00:06:05.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.335+0000 7f598e829640 1 -- 172.21.15.73:0/3148256982 wait complete. 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/4084025838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1520880569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3066299468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1539506988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/4084025838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1520880569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3066299468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.340 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1539506988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.341 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:05.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2126582808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 msgr2=0x7fbfd4108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2126582808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4108630 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc009ca0 tx=0x7fbfcc02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2126582808 shutdown_connections 2024-03-09T00:06:05.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2126582808 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4108630 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2126582808 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 0x7fbfd4101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2126582808 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2126582808 >> 172.21.15.73:0/2126582808 conn(0x7fbfd40fb780 msgr2=0x7fbfd40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2126582808 shutdown_connections 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2126582808 wait complete. 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 Processor -- start 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.451+0000 7fbfd9cc0640 1 -- start start 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4104180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 0x7fbfd41027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd410be80 con 0x7fbfd4101da0 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd410bd00 con 0x7fbfd41013e0 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfd410c000 con 0x7fbfd41001f0 2024-03-09T00:06:05.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32924/0 (socket says 172.21.15.73:32924) 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 -- 172.21.15.73:0/2699827744 learned_addr learned my addr 172.21.15.73:0/2699827744 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfcbfff640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 0x7fbfd41027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd37fe640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4104180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 -- 172.21.15.73:0/2699827744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 msgr2=0x7fbfd4104180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4104180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 -- 172.21.15.73:0/2699827744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 msgr2=0x7fbfd41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 0x7fbfd41027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 -- 172.21.15.73:0/2699827744 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd4103590 con 0x7fbfd4101da0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd37fe640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd3fff640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc02c700 tx=0x7fbfcc032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc039040 con 0x7fbfd4101da0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbfcc044070 con 0x7fbfd4101da0 2024-03-09T00:06:05.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfd41a1680 con 0x7fbfd4101da0 2024-03-09T00:06:05.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbfcc0042a0 con 0x7fbfd4101da0 2024-03-09T00:06:05.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.452+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfd41a1bd0 con 0x7fbfd4101da0 2024-03-09T00:06:05.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.453+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfd41a1e30 con 0x7fbfd4101da0 2024-03-09T00:06:05.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.454+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbfcc003970 con 0x7fbfd4101da0 2024-03-09T00:06:05.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.455+0000 7fbfd17fa640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 0x7fbfc00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.455+0000 7fbfd37fe640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 0x7fbfc00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.455+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbfcc0f1380 con 0x7fbfd4101da0 2024-03-09T00:06:05.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.456+0000 7fbfd37fe640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 0x7fbfc00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40058c0 tx=0x7fbfc4005d50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.460+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbfcc0be100 con 0x7fbfd4101da0 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: pgmap v351: 129 pgs: 129 active+clean; 639 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/4084025838' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1520880569' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3066299468' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:05.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1539506988' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:05.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.635+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fbfd41a2470 con 0x7fbfd4101da0 2024-03-09T00:06:05.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.635+0000 7fbfd17fa640 1 -- 172.21.15.73:0/2699827744 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v92) v1 ==== 100+0+11 (secure 0 0 0) 0x7fbfcc0bda00 con 0x7fbfd4101da0 2024-03-09T00:06:05.630 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 msgr2=0x7fbfc00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 0x7fbfc00a4cc0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fbfc40058c0 tx=0x7fbfc4005d50 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 msgr2=0x7fbfd4102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7fbfcc02c700 tx=0x7fbfcc032f20 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 shutdown_connections 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbfc00a2800 0x7fbfc00a4cc0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbfd4101da0 0x7fbfd4102d00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbfd41013e0 0x7fbfd41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 --2- 172.21.15.73:0/2699827744 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbfd41001f0 0x7fbfd4104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 >> 172.21.15.73:0/2699827744 conn(0x7fbfd40fb780 msgr2=0x7fbfd40fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.637+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 shutdown_connections 2024-03-09T00:06:05.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.638+0000 7fbfd9cc0640 1 -- 172.21.15.73:0/2699827744 wait complete. 2024-03-09T00:06:05.643 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:05.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- 172.21.15.73:0/22462643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c1046a0 msgr2=0x7f747c106a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- 172.21.15.73:0/22462643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c1046a0 0x7f747c106a90 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f7464009900 tx=0x7f746402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- 172.21.15.73:0/22462643 shutdown_connections 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- 172.21.15.73:0/22462643 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c106fd0 0x7f747c109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- 172.21.15.73:0/22462643 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c1046a0 0x7f747c106a90 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- 172.21.15.73:0/22462643 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f747c101d70 0x7f747c104160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- 172.21.15.73:0/22462643 >> 172.21.15.73:0/22462643 conn(0x7f747c0fb780 msgr2=0x7f747c0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- 172.21.15.73:0/22462643 shutdown_connections 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- 172.21.15.73:0/22462643 wait complete. 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 Processor -- start 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- start start 2024-03-09T00:06:05.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f747c101d70 0x7f747c101690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 0x7f747c0ffd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c10cf80 con 0x7f747c106fd0 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c10ce00 con 0x7f747c1046a0 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.753+0000 7f7481b6f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f747c10d100 con 0x7f747c101d70 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32934/0 (socket says 172.21.15.73:32934) 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 -- 172.21.15.73:0/2292943623 learned_addr learned my addr 172.21.15.73:0/2292943623 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f7473fff640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 0x7f747c0ffd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 -- 172.21.15.73:0/2292943623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f747c101d70 msgr2=0x7f747c101690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f747c101d70 0x7f747c101690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 -- 172.21.15.73:0/2292943623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 msgr2=0x7f747c0ffd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 0x7f747c0ffd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 -- 172.21.15.73:0/2292943623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f747c100b20 con 0x7f747c106fd0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f7473fff640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 0x7f747c0ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f747bfff640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f747400eb10 tx=0x7f747400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474004b20 con 0x7f747c106fd0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7474004cc0 con 0x7f747c106fd0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f747c1a1820 con 0x7f747c106fd0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7474010760 con 0x7f747c106fd0 2024-03-09T00:06:05.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.754+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f747c1a1cf0 con 0x7f747c106fd0 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.755+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f747c0fd270 con 0x7f747c106fd0 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.756+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f7474005020 con 0x7f747c106fd0 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.757+0000 7f74797fa640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 0x7f74680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.757+0000 7f747b7fe640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 0x7f74680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.757+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f74740cf4c0 con 0x7f747c106fd0 2024-03-09T00:06:05.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.758+0000 7f747b7fe640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 0x7f74680a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f746c005fd0 tx=0x7f746c005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:05.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.762+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f747409c1b0 con 0x7f747c106fd0 2024-03-09T00:06:05.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.939+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f747c061ef0 con 0x7f747c106fd0 2024-03-09T00:06:05.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.939+0000 7f74797fa640 1 -- 172.21.15.73:0/2292943623 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v92) v1 ==== 103+0+11 (secure 0 0 0) 0x7f747409bab0 con 0x7f747c106fd0 2024-03-09T00:06:05.934 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:05.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 msgr2=0x7f74680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 0x7f74680a4cc0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f746c005fd0 tx=0x7f746c005850 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 msgr2=0x7f747c100260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f747400eb10 tx=0x7f747400c2d0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 shutdown_connections 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f74680a2800 0x7f74680a4cc0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f747c106fd0 0x7f747c100260 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f747c1046a0 0x7f747c0ffd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 --2- 172.21.15.73:0/2292943623 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f747c101d70 0x7f747c101690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 >> 172.21.15.73:0/2292943623 conn(0x7f747c0fb780 msgr2=0x7f747c1028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 shutdown_connections 2024-03-09T00:06:05.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:05.941+0000 7f7481b6f640 1 -- 172.21.15.73:0/2292943623 wait complete. 2024-03-09T00:06:05.948 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 -- 172.21.15.73:0/3606838513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7341070b0 msgr2=0x7ff7341094a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/3606838513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7341070b0 0x7ff7341094a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009ca0 tx=0x7ff72c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 -- 172.21.15.73:0/3606838513 shutdown_connections 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/3606838513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7341070b0 0x7ff7341094a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/3606838513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff734104780 0x7ff734106b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/3606838513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734101e50 0x7ff734104240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 -- 172.21.15.73:0/3606838513 >> 172.21.15.73:0/3606838513 conn(0x7ff7340fb7a0 msgr2=0x7ff7340fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 -- 172.21.15.73:0/3606838513 shutdown_connections 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.054+0000 7ff73ba6c640 1 -- 172.21.15.73:0/3606838513 wait complete. 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 Processor -- start 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 -- start start 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 0x7ff734197b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7341070b0 0x7ff73419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73410cfe0 con 0x7ff734101e50 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73410ce60 con 0x7ff734104780 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73ba6c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73410d160 con 0x7ff7341070b0 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32958/0 (socket says 172.21.15.73:32958) 2024-03-09T00:06:06.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 -- 172.21.15.73:0/974965258 learned_addr learned my addr 172.21.15.73:0/974965258 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff73a009640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7341070b0 0x7ff73419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739007640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 0x7ff734197b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 -- 172.21.15.73:0/974965258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7341070b0 msgr2=0x7ff73419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7341070b0 0x7ff73419ebe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 -- 172.21.15.73:0/974965258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 msgr2=0x7ff734197b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.055+0000 7ff739808640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 0x7ff734197b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff739808640 1 -- 172.21.15.73:0/974965258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73419f250 con 0x7ff734101e50 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff739007640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 0x7ff734197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff739808640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff73000d950 tx=0x7ff73000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff730017030 con 0x7ff734101e50 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff730004020 con 0x7ff734101e50 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73419f540 con 0x7ff734101e50 2024-03-09T00:06:06.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7300047a0 con 0x7ff734101e50 2024-03-09T00:06:06.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.056+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73419fae0 con 0x7ff734101e50 2024-03-09T00:06:06.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.057+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7340fd290 con 0x7ff734101e50 2024-03-09T00:06:06.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.058+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7300041d0 con 0x7ff734101e50 2024-03-09T00:06:06.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.058+0000 7ff72affd640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 0x7ff7200a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.058+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff7300ce8f0 con 0x7ff734101e50 2024-03-09T00:06:06.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.058+0000 7ff739007640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 0x7ff7200a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.059+0000 7ff739007640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 0x7ff7200a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff734198b40 tx=0x7ff724005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.063+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff73009b5e0 con 0x7ff734101e50 2024-03-09T00:06:06.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.274+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff73402ce70 con 0x7ff734101e50 2024-03-09T00:06:06.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.275+0000 7ff72affd640 1 -- 172.21.15.73:0/974965258 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7ff73009aee0 con 0x7ff734101e50 2024-03-09T00:06:06.270 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:06.270 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:06.270 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:06.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.277+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 msgr2=0x7ff7200a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.277+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 0x7ff7200a4d40 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7ff734198b40 tx=0x7ff724005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 msgr2=0x7ff734197630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7ff73000d950 tx=0x7ff73000de10 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 shutdown_connections 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff7200a2880 0x7ff7200a4d40 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff7341070b0 0x7ff73419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff734104780 0x7ff734197b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 --2- 172.21.15.73:0/974965258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff734101e50 0x7ff734197630 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 >> 172.21.15.73:0/974965258 conn(0x7ff7340fb7a0 msgr2=0x7ff7341026f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 shutdown_connections 2024-03-09T00:06:06.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.278+0000 7ff73ba6c640 1 -- 172.21.15.73:0/974965258 wait complete. 2024-03-09T00:06:06.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: osdmap e92: 12 total, 12 up, 12 in 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3148256982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2699827744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2292943623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.284 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-09T00:06:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-09T00:06:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-09T00:06:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: osdmap e92: 12 total, 12 up, 12 in 2024-03-09T00:06:06.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-09T00:06:06.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3148256982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2699827744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2292943623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 -- 172.21.15.73:0/2598462925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf81046c0 msgr2=0x7fbaf8106ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/2598462925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf81046c0 0x7fbaf8106ab0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fbaec009ca0 tx=0x7fbaec02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 -- 172.21.15.73:0/2598462925 shutdown_connections 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/2598462925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 0x7fbaf8109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/2598462925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf81046c0 0x7fbaf8106ab0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/2598462925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf8101d90 0x7fbaf8104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 -- 172.21.15.73:0/2598462925 >> 172.21.15.73:0/2598462925 conn(0x7fbaf80fb780 msgr2=0x7fbaf80fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 -- 172.21.15.73:0/2598462925 shutdown_connections 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.391+0000 7fbafd2a0640 1 -- 172.21.15.73:0/2598462925 wait complete. 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 Processor -- start 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 -- start start 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf81046c0 0x7fbaf8197b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 0x7fbaf819ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf810cfa0 con 0x7fbaf8101d90 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf810ce20 con 0x7fbaf8106ff0 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbafd2a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbaf810d120 con 0x7fbaf81046c0 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbaf6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbaf6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32968/0 (socket says 172.21.15.73:32968) 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbaf6ffd640 1 -- 172.21.15.73:0/3509100331 learned_addr learned my addr 172.21.15.73:0/3509100331 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbaf77fe640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 0x7fbaf819ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.392+0000 7fbaf67fc640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf81046c0 0x7fbaf8197b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 -- 172.21.15.73:0/3509100331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf81046c0 msgr2=0x7fbaf8197b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf81046c0 0x7fbaf8197b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 -- 172.21.15.73:0/3509100331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 msgr2=0x7fbaf819ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 0x7fbaf819ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 -- 172.21.15.73:0/3509100331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbaf819f270 con 0x7fbaf8101d90 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbaf6ffd640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000c970 tx=0x7fbaf000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0007bb0 con 0x7fbaf8101d90 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbaf819f560 con 0x7fbaf8101d90 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbaf0007d50 con 0x7fbaf8101d90 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbaf0018800 con 0x7fbaf8101d90 2024-03-09T00:06:06.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.393+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbaf819fa80 con 0x7fbaf8101d90 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.394+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbaf80fd270 con 0x7fbaf8101d90 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.395+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbaf000f450 con 0x7fbaf8101d90 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.396+0000 7fbadffff640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 0x7fbae00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.396+0000 7fbaf67fc640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 0x7fbae00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.396+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbaf00cef00 con 0x7fbaf8101d90 2024-03-09T00:06:06.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.396+0000 7fbaf67fc640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 0x7fbae00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02c700 tx=0x7fbaec009360 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.400+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbaf0014030 con 0x7fbaf8101d90 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.5", "id": [8, 11]}]': finished 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.10", "id": [8, 11]}]': finished 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='mgr.14152 172.21.15.73:0/1389357722' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.16", "id": [2, 4]}]': finished 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: osdmap e92: 12 total, 12 up, 12 in 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-09T00:06:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3148256982' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2699827744' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2292943623' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:06.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.576+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbaf8061ef0 con 0x7fbaf8101d90 2024-03-09T00:06:06.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.578+0000 7fbadffff640 1 -- 172.21.15.73:0/3509100331 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7fbaf009bbf0 con 0x7fbaf8101d90 2024-03-09T00:06:06.573 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:06.573 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 msgr2=0x7fbae00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 0x7fbae00a4cc0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7fbaec02c700 tx=0x7fbaec009360 comp rx=0 tx=0).stop 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 msgr2=0x7fbaf8197650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fbaf000c970 tx=0x7fbaf000ce30 comp rx=0 tx=0).stop 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 shutdown_connections 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbae00a2800 0x7fbae00a4cc0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbaf8106ff0 0x7fbaf819ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbaf81046c0 0x7fbaf8197b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 --2- 172.21.15.73:0/3509100331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbaf8101d90 0x7fbaf8197650 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 >> 172.21.15.73:0/3509100331 conn(0x7fbaf80fb780 msgr2=0x7fbaf81027b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 shutdown_connections 2024-03-09T00:06:06.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.580+0000 7fbafd2a0640 1 -- 172.21.15.73:0/3509100331 wait complete. 2024-03-09T00:06:06.587 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:06.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 -- 172.21.15.73:0/3877204563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944101dc0 msgr2=0x7f8944108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 --2- 172.21.15.73:0/3877204563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944101dc0 0x7f8944108650 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7f893c009ca0 tx=0x7f893c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 -- 172.21.15.73:0/3877204563 shutdown_connections 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 --2- 172.21.15.73:0/3877204563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944101dc0 0x7f8944108650 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 --2- 172.21.15.73:0/3877204563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 --2- 172.21.15.73:0/3877204563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944100210 0x7f8944100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 -- 172.21.15.73:0/3877204563 >> 172.21.15.73:0/3877204563 conn(0x7f89440fb780 msgr2=0x7f89440fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 -- 172.21.15.73:0/3877204563 shutdown_connections 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.693+0000 7f894bdf3640 1 -- 172.21.15.73:0/3877204563 wait complete. 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 Processor -- start 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 -- start start 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944196a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944101dc0 0x7f894419da90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894410bea0 con 0x7f8944100210 2024-03-09T00:06:06.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894410bd20 con 0x7f8944101400 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894bdf3640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f894410c020 con 0x7f8944101dc0 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:32988/0 (socket says 172.21.15.73:32988) 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 -- 172.21.15.73:0/2044367615 learned_addr learned my addr 172.21.15.73:0/2044367615 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894a390640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944101dc0 0x7f894419da90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894938e640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944196a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 -- 172.21.15.73:0/2044367615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944101dc0 msgr2=0x7f894419da90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944101dc0 0x7f894419da90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 -- 172.21.15.73:0/2044367615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 msgr2=0x7f8944196a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944196a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f8949b8f640 1 -- 172.21.15.73:0/2044367615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894419e100 con 0x7f8944100210 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.694+0000 7f894938e640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f8949b8f640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f894000e9c0 tx=0x7f894000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f894000cd80 con 0x7f8944100210 2024-03-09T00:06:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f894419e3f0 con 0x7f8944100210 2024-03-09T00:06:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89400035f0 con 0x7f8944100210 2024-03-09T00:06:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8940010690 con 0x7f8944100210 2024-03-09T00:06:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.695+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f894419e8f0 con 0x7f8944100210 2024-03-09T00:06:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.696+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8944106f40 con 0x7f8944100210 2024-03-09T00:06:06.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.697+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f89400037a0 con 0x7f8944100210 2024-03-09T00:06:06.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.697+0000 7f893affd640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 0x7f89300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:06.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.698+0000 7f894938e640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 0x7f89300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:06.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.698+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f89400cf2f0 con 0x7f8944100210 2024-03-09T00:06:06.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.698+0000 7f894938e640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 0x7f89300a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89340058c0 tx=0x7f8934005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.701+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f894009bfe0 con 0x7f8944100210 2024-03-09T00:06:06.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.911+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8944107160 con 0x7f8944100210 2024-03-09T00:06:06.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.912+0000 7f893affd640 1 -- 172.21.15.73:0/2044367615 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f894009b8e0 con 0x7f8944100210 2024-03-09T00:06:06.911 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:06.911 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:06.913 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:06.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 msgr2=0x7f89300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 0x7f89300a4cc0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f89340058c0 tx=0x7f8934005850 comp rx=0 tx=0).stop 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 msgr2=0x7f89441964e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7f894000e9c0 tx=0x7f894000ee80 comp rx=0 tx=0).stop 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 shutdown_connections 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f89300a2800 0x7f89300a4cc0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8944101dc0 0x7f894419da90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8944101400 0x7f8944196a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 --2- 172.21.15.73:0/2044367615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8944100210 0x7f89441964e0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:06.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 >> 172.21.15.73:0/2044367615 conn(0x7f89440fb780 msgr2=0x7f89441046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:06.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 shutdown_connections 2024-03-09T00:06:06.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:06.914+0000 7f894bdf3640 1 -- 172.21.15.73:0/2044367615 wait complete. 2024-03-09T00:06:06.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-09T00:04:13.661086+0000', 'modified': '2024-03-09T00:04:39.321347+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6839/2754171892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 2754171892}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 2754171892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6839/3264474111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 3264474111}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 3264474111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-09T00:06:06.921 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-09T00:06:06.921 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 -- 172.21.15.73:0/3925127018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea81001f0 msgr2=0x7f2ea8100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 --2- 172.21.15.73:0/3925127018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea81001f0 0x7f2ea8100610 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7f2e98009900 tx=0x7f2e9802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 -- 172.21.15.73:0/3925127018 shutdown_connections 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 --2- 172.21.15.73:0/3925127018 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea8101da0 0x7f2ea8108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 --2- 172.21.15.73:0/3925127018 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 0x7f2ea8101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 --2- 172.21.15.73:0/3925127018 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea81001f0 0x7f2ea8100610 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 -- 172.21.15.73:0/3925127018 >> 172.21.15.73:0/3925127018 conn(0x7f2ea80fb780 msgr2=0x7f2ea80fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 -- 172.21.15.73:0/3925127018 shutdown_connections 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 -- 172.21.15.73:0/3925127018 wait complete. 2024-03-09T00:06:07.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.029+0000 7f2eae559640 1 Processor -- start 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 -- start start 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea81001f0 0x7f2ea8104180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 0x7f2ea81027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea810bea0 con 0x7f2ea8101da0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea810bd20 con 0x7f2ea81013e0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eae559640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea810c020 con 0x7f2ea81001f0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33014/0 (socket says 172.21.15.73:33014) 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 -- 172.21.15.73:0/4037735809 learned_addr learned my addr 172.21.15.73:0/4037735809 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea81001f0 0x7f2ea8104180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 0x7f2ea81027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 -- 172.21.15.73:0/4037735809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea81001f0 msgr2=0x7f2ea8104180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea81001f0 0x7f2ea8104180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 -- 172.21.15.73:0/4037735809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 msgr2=0x7f2ea81027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 0x7f2ea81027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 -- 172.21.15.73:0/4037735809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8103590 con 0x7f2ea8101da0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.030+0000 7f2eacaf6640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009fd0 tx=0x7f2ea000b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.031+0000 7f2ea57fa640 1 -- 172.21.15.73:0/4037735809 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea0017070 con 0x7f2ea8101da0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.031+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea81a17c0 con 0x7f2ea8101da0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.031+0000 7f2ea57fa640 1 -- 172.21.15.73:0/4037735809 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ea0003e50 con 0x7f2ea8101da0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.031+0000 7f2ea57fa640 1 -- 172.21.15.73:0/4037735809 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ea00106c0 con 0x7f2ea8101da0 2024-03-09T00:06:07.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.031+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea81a1d10 con 0x7f2ea8101da0 2024-03-09T00:06:07.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2ea57fa640 1 -- 172.21.15.73:0/4037735809 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2ea0003970 con 0x7f2ea8101da0 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 Processor -- start 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 -- start start 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6c9a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6ca20d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6bff7c0 con 0x556bb6c9aad0 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6ba8b50 con 0x556bb6c9fc40 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.033+0000 7f2eb120a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6bfb860 con 0x556bb6c9a0e0 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea57fa640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 0x7f2e900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33018/0 (socket says 172.21.15.73:33018) 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1056105485 learned_addr learned my addr 172.21.15.73:0/1056105485 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:07.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2eacaf6640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 0x7f2e900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6c9a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6ca20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1056105485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 msgr2=0x556bb6c9a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6c9a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1056105485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 msgr2=0x556bb6ca20d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6ca20d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1056105485 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bb6bbc9a0 con 0x556bb6c9aad0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea57fa640 1 -- 172.21.15.73:0/4037735809 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f2ea00cea90 con 0x7f2ea8101da0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2e98000c00 tx=0x7f2e98009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e53bd51be1620e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2eacaf6640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 0x7f2e900a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ea001d000 tx=0x7f2ea0069f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2e8e7fc640 1 -- 172.21.15.73:0/1056105485 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9802ca40 con 0x556bb6c9aad0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2e8e7fc640 1 -- 172.21.15.73:0/1056105485 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9802cbe0 con 0x556bb6c9aad0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.034+0000 7f2e8e7fc640 1 -- 172.21.15.73:0/1056105485 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98004920 con 0x556bb6c9aad0 2024-03-09T00:06:07.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- 172.21.15.73:0/1056105485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 msgr2=0x556bb6c9f5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7f2e98000c00 tx=0x7f2e98009610 comp rx=0 tx=0).stop 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- 172.21.15.73:0/1056105485 shutdown_connections 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6ca20d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6c9f5d0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1056105485 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6c9a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- 172.21.15.73:0/1056105485 >> 172.21.15.73:0/1056105485 conn(0x556bb6c62400 msgr2=0x556bb6c95a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- 172.21.15.73:0/1056105485 shutdown_connections 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- 172.21.15.73:0/1056105485 wait complete. 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.035+0000 7f2eb120a740 1 -- start start 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 Processor -- start 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 0x556bb6ca4a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6cabab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6ca2b90 con 0x556bb6c9a0e0 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6bbbf90 con 0x556bb6c9fc40 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6ca2d10 con 0x556bb6c9aad0 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2eb120a740 10 client.0 _ll_register_callbacks cb 0x556bb6bf7400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33034/0 (socket says 172.21.15.73:33034) 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 -- 172.21.15.73:0/3670723026 learned_addr learned my addr 172.21.15.73:0/3670723026 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 0x556bb6ca4a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 -- 172.21.15.73:0/3670723026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 msgr2=0x556bb6ca4a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 0x556bb6ca4a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 -- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 msgr2=0x556bb6cabab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6cabab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 -- 172.21.15.73:0/3670723026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bb69142d0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 0x556bb6ca4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0045c0 tx=0x7f2e9c0039c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a159a1fc01cf798a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.036+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c018070 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f2e9c01d660 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e9802cbe0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9c026050 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 10 client.15228 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556bb69de6b0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9802cbe0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2e9c012990 con 0x556bb6c9a0e0 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 10 client.15228 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:07.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 10 client.15228 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 20 client.15228 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6cb10e0 0x556bb6cb01e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 4 client.15228 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556bb6c0cd00 con 0x556bb6cb10e0 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.037+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6cb10e0 0x556bb6cb01e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.038+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6cb10e0 0x556bb6cb01e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3129b1330bbb0778 server_cookie=50e573a3cc81b2fc in_seq=0 out_seq=0 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.038+0000 7f2e877fe640 10 client.15228 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.038+0000 7f2e877fe640 1 -- 172.21.15.73:0/3670723026 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556bb6c0cd00 con 0x556bb6cb10e0 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.038+0000 7f2e877fe640 10 client.15228 handle_command_reply: tid=0 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 client.15228 shutdown 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 msgr2=0x556bb6ca4530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c0045c0 tx=0x7f2e9c0039c0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 shutdown_connections 2024-03-09T00:06:07.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6cb10e0 0x556bb6cb01e0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6cabab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9aad0 0x556bb6ca4a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2e877fe640 0 client.15228 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 --2- 172.21.15.73:0/3670723026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca4530 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 >> 172.21.15.73:0/3670723026 conn(0x556bb6c62400 msgr2=0x556bb6c94c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 shutdown_connections 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 1 -- 172.21.15.73:0/3670723026 wait complete. 2024-03-09T00:06:07.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.039+0000 7f2eb120a740 20 client.15228 trim_cache size 0 max 16384 2024-03-09T00:06:07.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.054+0000 7f2eb120a740 1 Processor -- start 2024-03-09T00:06:07.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.054+0000 7f2eb120a740 1 -- start start 2024-03-09T00:06:07.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9aad0 0x556bb6caba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9fc40 0x556bb6c991d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb0900 con 0x556bb6c9a0e0 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb0f60 con 0x556bb6c9aad0 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2eb120a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb15c0 con 0x556bb6c9fc40 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33044/0 (socket says 172.21.15.73:33044) 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 -- 172.21.15.73:0/2725063343 learned_addr learned my addr 172.21.15.73:0/2725063343 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 -- 172.21.15.73:0/2725063343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9fc40 msgr2=0x556bb6c991d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9fc40 0x556bb6c991d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 -- 172.21.15.73:0/2725063343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9aad0 msgr2=0x556bb6caba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9aad0 0x556bb6caba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 -- 172.21.15.73:0/2725063343 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556bb6c95ff0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c013f60 tx=0x7f2e9c013f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5529abca70f5a5fd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2e8dffb640 1 -- 172.21.15.73:0/2725063343 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c02e070 con 0x556bb6c9a0e0 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2e8dffb640 1 -- 172.21.15.73:0/2725063343 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9c004c80 con 0x556bb6c9a0e0 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.055+0000 7f2e8dffb640 1 -- 172.21.15.73:0/2725063343 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e9c018070 con 0x556bb6c9a0e0 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- 172.21.15.73:0/2725063343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 msgr2=0x556bb6c98c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7f2e9c013f60 tx=0x7f2e9c013f90 comp rx=0 tx=0).stop 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- 172.21.15.73:0/2725063343 shutdown_connections 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9fc40 0x556bb6c991d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9aad0 0x556bb6caba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 --2- 172.21.15.73:0/2725063343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9a0e0 0x556bb6c98c90 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- 172.21.15.73:0/2725063343 >> 172.21.15.73:0/2725063343 conn(0x556bb6c62400 msgr2=0x556bb6c9bf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- 172.21.15.73:0/2725063343 shutdown_connections 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- 172.21.15.73:0/2725063343 wait complete. 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.056+0000 7f2eb120a740 1 -- start start 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 Processor -- start 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6d06ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb2280 con 0x556bb6c9aad0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb1c20 con 0x556bb6c9fc40 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556bb6cb28e0 con 0x556bb6c9a0e0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 10 client.0 _ll_register_callbacks cb 0x7f2e98005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33060/0 (socket says 172.21.15.73:33060) 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1526677982 learned_addr learned my addr 172.21.15.73:0/1526677982 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6d06ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1526677982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 msgr2=0x556bb6ca8b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 msgr2=0x556bb6d06ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6d06ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 -- 172.21.15.73:0/1526677982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556bb6d072a0 con 0x556bb6c9aad0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2ea7fff640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x556bb6c9b540 tx=0x7f2e98009610 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b74795ae9c6ffe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:07.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2e877fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2e98034640 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f2e980347e0 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.057+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2e9c004c80 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2e9805d050 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 10 client.15234 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556bb6d085a0 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2e9c004c80 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2e9803fab0 con 0x556bb6c9aad0 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 10 client.15234 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 10 client.15234 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 20 client.15234 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6d0c880 0x556bb6d08770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:07.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 4 client.15234 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "status"}) v1 -- 0x556bb6c07050 con 0x556bb6d0c880 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.058+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6d0c880 0x556bb6d08770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2ea77fe640 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6d0c880 0x556bb6d08770 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6f48fbc303ef49c3 server_cookie=fb58c8e71a1d4 in_seq=0 out_seq=0 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2e877fe640 10 client.15234 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2e877fe640 1 -- 172.21.15.73:0/1526677982 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x556bb6c07050 con 0x556bb6d0c880 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2e877fe640 10 client.15234 handle_command_reply: tid=0 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 client.15234 shutdown 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 msgr2=0x556bb6ca9040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x556bb6c9b540 tx=0x7f2e98009610 comp rx=0 tx=0).stop 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 shutdown_connections 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556bb6d0c880 0x556bb6d08770 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556bb6c9fc40 0x556bb6d06ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2e877fe640 0 client.15234 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556bb6c9aad0 0x556bb6ca9040 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 --2- 172.21.15.73:0/1526677982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556bb6c9a0e0 0x556bb6ca8b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 >> 172.21.15.73:0/1526677982 conn(0x556bb6c62400 msgr2=0x556bb6c94400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 shutdown_connections 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 1 -- 172.21.15.73:0/1526677982 wait complete. 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eb120a740 20 client.15234 trim_cache size 0 max 16384 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout: "cluster_fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout: "whoami": 0, 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout: "id": 24515, 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout: "want_state": "up:active", 2024-03-09T00:06:07.062 INFO:teuthology.orchestra.run.smithi073.stdout: "state": "up:active", 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "fs_name": "cephfs", 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "rank_uptime": 112.737607732, 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "mdsmap_epoch": 20, 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch": 93, 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch_barrier": 89, 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout: "uptime": 118.14410450600001 2024-03-09T00:06:07.063 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:06:07.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 msgr2=0x7f2e900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 0x7f2e900a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f2ea001d000 tx=0x7f2ea0069f90 comp rx=0 tx=0).stop 2024-03-09T00:06:07.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 msgr2=0x7f2ea8102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.059+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7f2ea0009fd0 tx=0x7f2ea000b260 comp rx=0 tx=0).stop 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 shutdown_connections 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f2e900a2800 0x7f2e900a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2ea8101da0 0x7f2ea8102d00 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2ea81013e0 0x7f2ea81027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 --2- 172.21.15.73:0/4037735809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2ea81001f0 0x7f2ea8104180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 >> 172.21.15.73:0/4037735809 conn(0x7f2ea80fb780 msgr2=0x7f2ea8068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 shutdown_connections 2024-03-09T00:06:07.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:07.060+0000 7f2eae559640 1 -- 172.21.15.73:0/4037735809 wait complete. 2024-03-09T00:06:07.066 INFO:tasks.cephadm:Setup complete, yielding 2024-03-09T00:06:07.067 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-09T00:06:07.088 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:06:07.088 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph orch status' 2024-03-09T00:06:07.325 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:07 smithi179 ceph-mon[36637]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-09T00:06:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:07 smithi179 ceph-mon[36637]: osdmap e93: 12 total, 12 up, 12 in 2024-03-09T00:06:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/974965258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3509100331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2044367615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:07 smithi073 ceph-mon[32309]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-09T00:06:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:07 smithi073 ceph-mon[32309]: osdmap e93: 12 total, 12 up, 12 in 2024-03-09T00:06:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/974965258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3509100331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2044367615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:07 smithi102 ceph-mon[36106]: pgmap v353: 129 pgs: 129 active+clean; 640 KiB data, 379 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 307 B/s wr, 2 op/s 2024-03-09T00:06:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:07 smithi102 ceph-mon[36106]: osdmap e93: 12 total, 12 up, 12 in 2024-03-09T00:06:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/974965258' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3509100331' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:07.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2044367615' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:09.087 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:09 smithi073 ceph-mon[32309]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:09.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/960341030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc00ffe70 msgr2=0x7f1fc01002d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/960341030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc00ffe70 0x7f1fc01002d0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f1fac009990 tx=0x7f1fac009ca0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/960341030 shutdown_connections 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/960341030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 0x7f1fc010a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/960341030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc01033f0 0x7f1fc0103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/960341030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc00ffe70 0x7f1fc01002d0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/960341030 >> 172.21.15.73:0/960341030 conn(0x7f1fc00fba50 msgr2=0x7f1fc00fde70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/960341030 shutdown_connections 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/960341030 wait complete. 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 Processor -- start 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.190+0000 7f1fc6f3d700 1 -- start start 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 0x7f1fc0197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:09.187 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 0x7f1fc01984a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc010d9d0 con 0x7f1fc01033f0 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc010d850 con 0x7f1fc00ffe70 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc6f3d700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc010db50 con 0x7f1fc0100810 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41610/0 (socket says 172.21.15.73:41610) 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 -- 172.21.15.73:0/3365946295 learned_addr learned my addr 172.21.15.73:0/3365946295 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc4cd9700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 0x7f1fc0197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fbffff700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 0x7f1fc01984a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 -- 172.21.15.73:0/3365946295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 msgr2=0x7f1fc01984a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:09.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 0x7f1fc01984a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 -- 172.21.15.73:0/3365946295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 msgr2=0x7f1fc0197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 0x7f1fc0197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 -- 172.21.15.73:0/3365946295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc019fbc0 con 0x7f1fc01033f0 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc4cd9700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 0x7f1fc0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fc54da700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007da0 tx=0x7f1fb800a340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.191+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8016020 con 0x7f1fc01033f0 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.192+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc019feb0 con 0x7f1fc01033f0 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.192+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1fb800ad00 con 0x7f1fc01033f0 2024-03-09T00:06:09.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.192+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1fb8011620 con 0x7f1fc01033f0 2024-03-09T00:06:09.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.192+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc01a0310 con 0x7f1fc01033f0 2024-03-09T00:06:09.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.193+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fc0068aa0 con 0x7f1fc01033f0 2024-03-09T00:06:09.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.194+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f1fb8019020 con 0x7f1fc01033f0 2024-03-09T00:06:09.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.194+0000 7f1fbdffb700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 0x7f1fb00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:09.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.194+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f1fb80ced70 con 0x7f1fc01033f0 2024-03-09T00:06:09.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.194+0000 7f1fc4cd9700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 0x7f1fb00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:09.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.195+0000 7f1fc4cd9700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 0x7f1fb00a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00697b0 tx=0x7f1fb400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:09.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.199+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1fb8020020 con 0x7f1fc01033f0 2024-03-09T00:06:09.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:09 smithi179 ceph-mon[36637]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:09.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.411+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7f1fc0062220 con 0x7f1fb00a2890 2024-03-09T00:06:09.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.413+0000 7f1fbdffb700 1 -- 172.21.15.73:0/3365946295 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7f1fc0062220 con 0x7f1fb00a2890 2024-03-09T00:06:09.408 INFO:teuthology.orchestra.run.smithi073.stdout:Backend: cephadm 2024-03-09T00:06:09.408 INFO:teuthology.orchestra.run.smithi073.stdout:Available: Yes 2024-03-09T00:06:09.408 INFO:teuthology.orchestra.run.smithi073.stdout:Paused: No 2024-03-09T00:06:09.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 msgr2=0x7f1fb00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 0x7f1fb00a4d50 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f1fc00697b0 tx=0x7f1fb400b410 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 msgr2=0x7f1fc019f520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f1fb8007da0 tx=0x7f1fb800a340 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 shutdown_connections 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1fc01033f0 0x7f1fc019f520 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f1fb00a2890 0x7f1fb00a4d50 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1fc00ffe70 0x7f1fc0197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 --2- 172.21.15.73:0/3365946295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1fc0100810 0x7f1fc01984a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 >> 172.21.15.73:0/3365946295 conn(0x7f1fc00fba50 msgr2=0x7f1fc0106430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 shutdown_connections 2024-03-09T00:06:09.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:09.415+0000 7f1fc6f3d700 1 -- 172.21.15.73:0/3365946295 wait complete. 2024-03-09T00:06:09.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:09 smithi102 ceph-mon[36106]: pgmap v355: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 383 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:10.190 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-09T00:06:10.431 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:11.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:11 smithi073 ceph-mon[32309]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:11.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:11 smithi073 ceph-mon[32309]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:11.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:11 smithi179 ceph-mon[36637]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:11.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:11 smithi179 ceph-mon[36637]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:11.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:11 smithi102 ceph-mon[36106]: from='client.15240 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:11.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:11 smithi102 ceph-mon[36106]: pgmap v356: 129 pgs: 1 activating, 128 active+clean; 640 KiB data, 387 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 -- 172.21.15.73:0/1182345408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 msgr2=0x7fc84c109f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 --2- 172.21.15.73:0/1182345408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c109f70 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fc84400b0a0 tx=0x7fc84400b3b0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 -- 172.21.15.73:0/1182345408 shutdown_connections 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 --2- 172.21.15.73:0/1182345408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c109f70 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 --2- 172.21.15.73:0/1182345408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c104c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 --2- 172.21.15.73:0/1182345408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 0x7fc84c1075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 -- 172.21.15.73:0/1182345408 >> 172.21.15.73:0/1182345408 conn(0x7fc84c0fc2b0 msgr2=0x7fc84c0fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:12.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 -- 172.21.15.73:0/1182345408 shutdown_connections 2024-03-09T00:06:12.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 -- 172.21.15.73:0/1182345408 wait complete. 2024-03-09T00:06:12.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.343+0000 7fc851b38700 1 Processor -- start 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 -- start start 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 0x7fc84c100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c10d9e0 con 0x7fc84c107b00 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c10d860 con 0x7fc84c1051d0 2024-03-09T00:06:12.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc851b38700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc84c10db60 con 0x7fc84c1028a0 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84b7fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84b7fe700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:52828/0 (socket says 172.21.15.73:52828) 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84b7fe700 1 -- 172.21.15.73:0/862439734 learned_addr learned my addr 172.21.15.73:0/862439734 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84affd700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 0x7fc84c100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 -- 172.21.15.73:0/862439734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 msgr2=0x7fc84c102160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 -- 172.21.15.73:0/862439734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 msgr2=0x7fc84c100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 0x7fc84c100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 -- 172.21.15.73:0/862439734 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c1015a0 con 0x7fc84c107b00 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84b7fe700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.344+0000 7fc84bfff700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c100d90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc844009bc0 tx=0x7fc844004d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.345+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc844021070 con 0x7fc84c107b00 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.345+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc84c1a2130 con 0x7fc84c107b00 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.345+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc844007b80 con 0x7fc84c107b00 2024-03-09T00:06:12.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.345+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84401d450 con 0x7fc84c107b00 2024-03-09T00:06:12.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.345+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc84c1a2630 con 0x7fc84c107b00 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.346+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc84c068aa0 con 0x7fc84c107b00 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.347+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc844004000 con 0x7fc84c107b00 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.348+0000 7fc848ff9700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.348+0000 7fc84b7fe700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.348+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc84400f020 con 0x7fc84c107b00 2024-03-09T00:06:12.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.348+0000 7fc84b7fe700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 0x7fc8340a4d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc83c011590 tx=0x7fc83c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:12.346 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.351+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc84402a020 con 0x7fc84c107b00 2024-03-09T00:06:12.548 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.553+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fc84c062220 con 0x7fc8340a2890 2024-03-09T00:06:12.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.566+0000 7fc848ff9700 1 -- 172.21.15.73:0/862439734 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4092 (secure 0 0 0) 0x7fc84c062220 con 0x7fc8340a2890 2024-03-09T00:06:12.561 INFO:teuthology.orchestra.run.smithi073.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-09T00:06:12.561 INFO:teuthology.orchestra.run.smithi073.stdout:mds.a smithi073 running (2m) 25s ago 2m 13.8M - 19.0.0-1965-g7d63482a a272146cb810 7b33c054aa0e 2024-03-09T00:06:12.561 INFO:teuthology.orchestra.run.smithi073.stdout:mds.b smithi102 running (2m) 57s ago 2m 27.9M - 19.0.0-1965-g7d63482a a272146cb810 bfe0134cfd31 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.c smithi179 running (2m) 57s ago 2m 13.9M - 19.0.0-1965-g7d63482a a272146cb810 6dbee69e5944 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.d smithi073 running (2m) 25s ago 2m 14.0M - 19.0.0-1965-g7d63482a a272146cb810 c3660eafc5f4 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.e smithi102 running (2m) 57s ago 2m 15.3M - 19.0.0-1965-g7d63482a a272146cb810 fde9e6bd001c 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.f smithi179 running (2m) 57s ago 2m 13.9M - 19.0.0-1965-g7d63482a a272146cb810 0de693ebc82c 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.g smithi073 running (2m) 25s ago 2m 12.8M - 19.0.0-1965-g7d63482a a272146cb810 aa805f636c59 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.h smithi102 running (2m) 57s ago 2m 13.9M - 19.0.0-1965-g7d63482a a272146cb810 937a46e83c3a 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.i smithi179 running (2m) 57s ago 2m 16.9M - 19.0.0-1965-g7d63482a a272146cb810 643e59857f21 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.j smithi073 running (2m) 25s ago 2m 15.4M - 19.0.0-1965-g7d63482a a272146cb810 070b7a4d5e5d 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.k smithi102 running (2m) 57s ago 2m 16.3M - 19.0.0-1965-g7d63482a a272146cb810 492fb651b03c 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mds.l smithi179 running (2m) 57s ago 2m 16.2M - 19.0.0-1965-g7d63482a a272146cb810 8220a0c5a91f 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.x smithi073 *:9283,8765 running (9m) 25s ago 9m 513M - 19.0.0-1965-g7d63482a a272146cb810 3d5bbc20058d 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.y smithi102 *:8443,8765 running (7m) 57s ago 7m 452M - 19.0.0-1965-g7d63482a a272146cb810 5139b0975ba2 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mgr.z smithi179 *:8443,8765 running (7m) 57s ago 7m 453M - 19.0.0-1965-g7d63482a a272146cb810 b45a969a2250 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mon.a smithi073 running (9m) 25s ago 10m 50.2M 2048M 19.0.0-1965-g7d63482a a272146cb810 7374c2d99018 2024-03-09T00:06:12.562 INFO:teuthology.orchestra.run.smithi073.stdout:mon.b smithi102 running (8m) 57s ago 8m 39.9M 2048M 19.0.0-1965-g7d63482a a272146cb810 feeab1859012 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:mon.c smithi179 running (8m) 57s ago 8m 40.3M 2048M 19.0.0-1965-g7d63482a a272146cb810 a1f882149fa6 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 smithi073 running (7m) 25s ago 7m 45.5M 4272M 19.0.0-1965-g7d63482a a272146cb810 75e4558d6757 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 smithi102 running (7m) 57s ago 7m 54.2M 4270M 19.0.0-1965-g7d63482a a272146cb810 0017561d8b1c 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 smithi179 running (6m) 57s ago 6m 46.5M 4270M 19.0.0-1965-g7d63482a a272146cb810 e04dd2f6fb3b 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 smithi073 running (6m) 25s ago 6m 44.0M 4272M 19.0.0-1965-g7d63482a a272146cb810 0eced198d432 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 smithi102 running (5m) 57s ago 5m 49.7M 4270M 19.0.0-1965-g7d63482a a272146cb810 c255b28bec77 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 smithi179 running (5m) 57s ago 5m 56.2M 4270M 19.0.0-1965-g7d63482a a272146cb810 72c095c36fff 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 smithi073 running (5m) 25s ago 5m 50.2M 4272M 19.0.0-1965-g7d63482a a272146cb810 51a06e528129 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 smithi102 running (4m) 57s ago 4m 55.1M 4270M 19.0.0-1965-g7d63482a a272146cb810 e0cb21bb69f5 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.8 smithi179 running (4m) 57s ago 4m 43.8M 4270M 19.0.0-1965-g7d63482a a272146cb810 9576566eae86 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.9 smithi073 running (3m) 25s ago 3m 52.9M 4272M 19.0.0-1965-g7d63482a a272146cb810 f305c848a3c6 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.10 smithi102 running (3m) 57s ago 3m 50.4M 4270M 19.0.0-1965-g7d63482a a272146cb810 e6f06c4ad7e9 2024-03-09T00:06:12.563 INFO:teuthology.orchestra.run.smithi073.stdout:osd.11 smithi179 running (2m) 57s ago 2m 49.1M 4270M 19.0.0-1965-g7d63482a a272146cb810 8c614f05dd61 2024-03-09T00:06:12.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.568+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 msgr2=0x7fc8340a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:12.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 0x7fc8340a4d50 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7fc83c011590 tx=0x7fc83c009260 comp rx=0 tx=0).stop 2024-03-09T00:06:12.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 msgr2=0x7fc84c100d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:12.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c100d90 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7fc844009bc0 tx=0x7fc844004d00 comp rx=0 tx=0).stop 2024-03-09T00:06:12.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 shutdown_connections 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc84c107b00 0x7fc84c100d90 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc8340a2890 0x7fc8340a4d50 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc84c1028a0 0x7fc84c102160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 --2- 172.21.15.73:0/862439734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc84c1051d0 0x7fc84c100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 >> 172.21.15.73:0/862439734 conn(0x7fc84c0fc2b0 msgr2=0x7fc84c103200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 shutdown_connections 2024-03-09T00:06:12.565 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:12.569+0000 7fc851b38700 1 -- 172.21.15.73:0/862439734 wait complete. 2024-03-09T00:06:13.242 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:13 smithi073 ceph-mon[32309]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:13.269 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-09T00:06:13.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:13 smithi179 ceph-mon[36637]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:13.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:13 smithi102 ceph-mon[36106]: pgmap v357: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 383 B/s wr, 3 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:13.509 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:14.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:14 smithi073 ceph-mon[32309]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:14.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:14 smithi179 ceph-mon[36637]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:14.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:14 smithi102 ceph-mon[36106]: from='client.15246 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:15.381 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:15 smithi073 ceph-mon[32309]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:15.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 -- 172.21.15.73:0/1958652445 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d44101f20 msgr2=0x7f4d44102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:15.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/1958652445 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d44101f20 0x7f4d44102380 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7f4d400099c0 tx=0x7f4d40009cd0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 -- 172.21.15.73:0/1958652445 shutdown_connections 2024-03-09T00:06:15.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/1958652445 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d441028c0 0x7f4d44109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/1958652445 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 0x7f4d44101140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/1958652445 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d44101f20 0x7f4d44102380 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 -- 172.21.15.73:0/1958652445 >> 172.21.15.73:0/1958652445 conn(0x7f4d440fc2e0 msgr2=0x7f4d440fe700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 -- 172.21.15.73:0/1958652445 shutdown_connections 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.386+0000 7f4d4c00b700 1 -- 172.21.15.73:0/1958652445 wait complete. 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 Processor -- start 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 -- start start 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 0x7f4d4406b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d44101f20 0x7f4d4419cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4410c980 con 0x7f4d441028c0 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4410c800 con 0x7f4d44101f20 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4c00b700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d4410cb00 con 0x7f4d44100d20 2024-03-09T00:06:15.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4a5a8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4a5a8700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41636/0 (socket says 172.21.15.73:41636) 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d4a5a8700 1 -- 172.21.15.73:0/4244796273 learned_addr learned my addr 172.21.15.73:0/4244796273 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.387+0000 7f4d495a6700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d44101f20 0x7f4d4419cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d49da7700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 0x7f4d4406b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 -- 172.21.15.73:0/4244796273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 msgr2=0x7f4d4406b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 0x7f4d4406b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 -- 172.21.15.73:0/4244796273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d44101f20 msgr2=0x7f4d4419cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d44101f20 0x7f4d4419cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 -- 172.21.15.73:0/4244796273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d4419fd00 con 0x7f4d441028c0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4a5a8700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c007970 tx=0x7f4d3c007c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c00d5e0 con 0x7f4d441028c0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4d3c00dc60 con 0x7f4d441028c0 2024-03-09T00:06:15.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d4419fff0 con 0x7f4d441028c0 2024-03-09T00:06:15.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4d3c013e50 con 0x7f4d441028c0 2024-03-09T00:06:15.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.388+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d441a0520 con 0x7f4d441028c0 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.389+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d44068aa0 con 0x7f4d441028c0 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.391+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f4d3c01a020 con 0x7f4d441028c0 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.391+0000 7f4d3affd700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 0x7f4d300a4ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.392+0000 7f4d49da7700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 0x7f4d300a4ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.392+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f4d3c0ceba0 con 0x7f4d441028c0 2024-03-09T00:06:15.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.392+0000 7f4d49da7700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 0x7f4d300a4ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d44101d80 tx=0x7f4d34009190 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:15.390 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.395+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4d3c010070 con 0x7f4d441028c0 2024-03-09T00:06:15.410 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:15 smithi102 ceph-mon[36106]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:15.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:15 smithi179 ceph-mon[36637]: pgmap v358: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 575 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:15.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.599+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d44062220 con 0x7f4d300a2a30 2024-03-09T00:06:15.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.605+0000 7f4d3affd700 1 -- 172.21.15.73:0/4244796273 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f4d44062220 con 0x7f4d300a2a30 2024-03-09T00:06:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-09T00:06:15.599 INFO:teuthology.orchestra.run.smithi073.stdout:mds.all 12/12 60s ago 2m smithi073=a;smithi073=d;smithi073=g;smithi073=j;smithi102=b;smithi102=e;smithi102=h;smithi102=k;smithi179=c;smithi179=f;smithi179=i;smithi179=l;count:12 2024-03-09T00:06:15.600 INFO:teuthology.orchestra.run.smithi073.stdout:mgr 3/3 60s ago 8m smithi073=x;smithi102=y;smithi179=z;count:3 2024-03-09T00:06:15.600 INFO:teuthology.orchestra.run.smithi073.stdout:mon 3/3 60s ago 8m smithi073:172.21.15.73=a;smithi102:172.21.15.102=b;smithi179:172.21.15.179=c;count:3 2024-03-09T00:06:15.600 INFO:teuthology.orchestra.run.smithi073.stdout:osd 12 60s ago - 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 msgr2=0x7f4d300a4ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 0x7f4d300a4ef0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f4d44101d80 tx=0x7f4d34009190 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 msgr2=0x7f4d4419f430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f4d3c007970 tx=0x7f4d3c007c80 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 shutdown_connections 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4d441028c0 0x7f4d4419f430 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f4d300a2a30 0x7f4d300a4ef0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4d44100d20 0x7f4d4406b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 --2- 172.21.15.73:0/4244796273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4d44101f20 0x7f4d4419cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 >> 172.21.15.73:0/4244796273 conn(0x7f4d440fc2e0 msgr2=0x7f4d44105220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:15.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 shutdown_connections 2024-03-09T00:06:15.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:15.607+0000 7f4d4c00b700 1 -- 172.21.15.73:0/4244796273 wait complete. 2024-03-09T00:06:16.306 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-09T00:06:16.549 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:17 smithi073 ceph-mon[32309]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:17.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:17 smithi073 ceph-mon[32309]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:17.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:17 smithi102 ceph-mon[36106]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:17.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:17 smithi102 ceph-mon[36106]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:17.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:17 smithi179 ceph-mon[36637]: from='client.15252 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:17.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:17 smithi179 ceph-mon[36637]: pgmap v359: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 0 B/s wr, 2 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 -- 172.21.15.73:0/210825754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c100d00 msgr2=0x7f0b3c101120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 --2- 172.21.15.73:0/210825754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c100d00 0x7f0b3c101120 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0b3400b0d0 tx=0x7f0b3400b3e0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 -- 172.21.15.73:0/210825754 shutdown_connections 2024-03-09T00:06:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 --2- 172.21.15.73:0/210825754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c100d00 0x7f0b3c101120 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 --2- 172.21.15.73:0/210825754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c109150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 --2- 172.21.15.73:0/210825754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 0x7f0b3c102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 -- 172.21.15.73:0/210825754 >> 172.21.15.73:0/210825754 conn(0x7f0b3c0fc2c0 msgr2=0x7f0b3c0fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:18.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 -- 172.21.15.73:0/210825754 shutdown_connections 2024-03-09T00:06:18.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.300+0000 7f0b40820700 1 -- 172.21.15.73:0/210825754 wait complete. 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 Processor -- start 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 -- start start 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 0x7f0b3c196e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 0x7f0b3c1973d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c10c8e0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c10c760 con 0x7f0b3c101f00 2024-03-09T00:06:18.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b40820700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b3c10ca60 con 0x7f0b3c100d00 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b3ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b3a59c700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 0x7f0b3c196e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b3ad9d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41652/0 (socket says 172.21.15.73:41652) 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b3ad9d700 1 -- 172.21.15.73:0/2147814801 learned_addr learned my addr 172.21.15.73:0/2147814801 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.301+0000 7f0b39d9b700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 0x7f0b3c1973d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 -- 172.21.15.73:0/2147814801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 msgr2=0x7f0b3c196e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 0x7f0b3c196e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 -- 172.21.15.73:0/2147814801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 msgr2=0x7f0b3c1973d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 0x7f0b3c1973d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 -- 172.21.15.73:0/2147814801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b3c19eaf0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3a59c700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 0x7f0b3c196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b3ad9d700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0096c0 tx=0x7f0b2c0099d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:18.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c004830 con 0x7f0b3c1028a0 2024-03-09T00:06:18.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b3c06a500 con 0x7f0b3c1028a0 2024-03-09T00:06:18.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b2c005020 con 0x7f0b3c1028a0 2024-03-09T00:06:18.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2c0167e0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.302+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b3c06aa90 con 0x7f0b3c1028a0 2024-03-09T00:06:18.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.303+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b3c068aa0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.304+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0b2c0049d0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.305+0000 7f0b337fe700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 0x7f0b280a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:18.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.305+0000 7f0b3a59c700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 0x7f0b280a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:18.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.305+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0b2c0ce350 con 0x7f0b3c1028a0 2024-03-09T00:06:18.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.305+0000 7f0b3a59c700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 0x7f0b280a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009fd0 tx=0x7f0b3400b040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:18.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.309+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0b2c09b0e0 con 0x7f0b3c1028a0 2024-03-09T00:06:18.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.510+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b3c06acf0 con 0x7f0b280a2890 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.511+0000 7f0b337fe700 1 -- 172.21.15.73:0/2147814801 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f0b3c06acf0 con 0x7f0b280a2890 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stdout:HOST ADDR LABELS STATUS 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 172.21.15.73 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stdout:smithi102 172.21.15.102 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stdout:smithi179 172.21.15.179 2024-03-09T00:06:18.506 INFO:teuthology.orchestra.run.smithi073.stdout:3 hosts in cluster 2024-03-09T00:06:18.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 msgr2=0x7f0b280a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:18.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 0x7f0b280a4d50 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f0b34009fd0 tx=0x7f0b3400b040 comp rx=0 tx=0).stop 2024-03-09T00:06:18.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 msgr2=0x7f0b3c19e450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:18.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f0b2c0096c0 tx=0x7f0b2c0099d0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 shutdown_connections 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b3c1028a0 0x7f0b3c19e450 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0b280a2890 0x7f0b280a4d50 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b3c100d00 0x7f0b3c196e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 --2- 172.21.15.73:0/2147814801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b3c101f00 0x7f0b3c1973d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.513+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 >> 172.21.15.73:0/2147814801 conn(0x7f0b3c0fc2c0 msgr2=0x7f0b3c1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.514+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 shutdown_connections 2024-03-09T00:06:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:18.514+0000 7f0b40820700 1 -- 172.21.15.73:0/2147814801 wait complete. 2024-03-09T00:06:19.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:19 smithi073 ceph-mon[32309]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:19.200 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-09T00:06:19.438 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:19.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:19 smithi102 ceph-mon[36106]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:19.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:19 smithi179 ceph-mon[36637]: pgmap v360: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 861 B/s rd, 0 B/s wr, 1 op/s; 0 B/s, 0 objects/s recovering 2024-03-09T00:06:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:20 smithi073 ceph-mon[32309]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:20.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:20 smithi102 ceph-mon[36106]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:20.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:20 smithi179 ceph-mon[36637]: from='client.15258 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:21.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:21 smithi073 ceph-mon[32309]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 -- 172.21.15.73:0/4193706816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8100d10 msgr2=0x7febc8101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 --2- 172.21.15.73:0/4193706816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8100d10 0x7febc8101130 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7febc000b0d0 tx=0x7febc000b3e0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 -- 172.21.15.73:0/4193706816 shutdown_connections 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 --2- 172.21.15.73:0/4193706816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8100d10 0x7febc8101130 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 --2- 172.21.15.73:0/4193706816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc81028b0 0x7febc8109160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 --2- 172.21.15.73:0/4193706816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc8101f10 0x7febc8102370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 -- 172.21.15.73:0/4193706816 >> 172.21.15.73:0/4193706816 conn(0x7febc80fc2b0 msgr2=0x7febc80fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 -- 172.21.15.73:0/4193706816 shutdown_connections 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.169+0000 7febcf508700 1 -- 172.21.15.73:0/4193706816 wait complete. 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 Processor -- start 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 -- start start 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 0x7febc8104ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc81028b0 0x7febc8103830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc810c8f0 con 0x7febc8101f10 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc810c770 con 0x7febc81028b0 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcf508700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febc810ca70 con 0x7febc8100d10 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:21.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49248/0 (socket says 172.21.15.73:49248) 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 -- 172.21.15.73:0/3186215942 learned_addr learned my addr 172.21.15.73:0/3186215942 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcd2a4700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 0x7febc8104ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcdaa5700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc81028b0 0x7febc8103830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 -- 172.21.15.73:0/3186215942 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 msgr2=0x7febc8104ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 0x7febc8104ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 -- 172.21.15.73:0/3186215942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc81028b0 msgr2=0x7febc8103830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc81028b0 0x7febc8103830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 -- 172.21.15.73:0/3186215942 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febc81040e0 con 0x7febc8101f10 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febcd2a4700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 0x7febc8104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.170+0000 7febccaa3700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7febbc00d950 tx=0x7febbc00dd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.171+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc016070 con 0x7febc8101f10 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.171+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febc81a21f0 con 0x7febc8101f10 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.171+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7febbc009b40 con 0x7febc8101f10 2024-03-09T00:06:21.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.171+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7febbc00f620 con 0x7febc8101f10 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.171+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febc81a26e0 con 0x7febc8101f10 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.172+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febc8068aa0 con 0x7febc8101f10 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.173+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7febbc019070 con 0x7febc8101f10 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.173+0000 7febc67fc700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 0x7febb00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.173+0000 7febcd2a4700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 0x7febb00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.173+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7febbc0ceea0 con 0x7febc8101f10 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.174+0000 7febcd2a4700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 0x7febb00a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7febc0009850 tx=0x7febc0009780 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:21.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.177+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7febbc09bb90 con 0x7febc8101f10 2024-03-09T00:06:21.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.384+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7febc80fe4a0 con 0x7febb00a2890 2024-03-09T00:06:21.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.386+0000 7febc67fc700 1 -- 172.21.15.73:0/3186215942 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7febc80fe4a0 con 0x7febb00a2890 2024-03-09T00:06:21.381 INFO:teuthology.orchestra.run.smithi073.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-09T00:06:21.381 INFO:teuthology.orchestra.run.smithi073.stdout:smithi073 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT62040115400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-09T00:06:21.381 INFO:teuthology.orchestra.run.smithi073.stdout:smithi102 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400VK400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-09T00:06:21.381 INFO:teuthology.orchestra.run.smithi073.stdout:smithi179 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6303009J400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 msgr2=0x7febb00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 0x7febb00a4d50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7febc0009850 tx=0x7febc0009780 comp rx=0 tx=0).stop 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 msgr2=0x7febc81032f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7febbc00d950 tx=0x7febbc00dd10 comp rx=0 tx=0).stop 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 shutdown_connections 2024-03-09T00:06:21.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7febb00a2890 0x7febb00a4d50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7febc81028b0 0x7febc8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7febc8100d10 0x7febc8104ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 --2- 172.21.15.73:0/3186215942 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7febc8101f10 0x7febc81032f0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 >> 172.21.15.73:0/3186215942 conn(0x7febc80fc2b0 msgr2=0x7febc80fdd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 shutdown_connections 2024-03-09T00:06:21.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:21.388+0000 7febcf508700 1 -- 172.21.15.73:0/3186215942 wait complete. 2024-03-09T00:06:21.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:21 smithi102 ceph-mon[36106]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-09T00:06:21.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:21 smithi179 ceph-mon[36637]: pgmap v361: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 0 B/s wr, 2 op/s 2024-03-09T00:06:22.078 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-09T00:06:22.128 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:06:22.128 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-09T00:06:22.372 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:23.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:23 smithi073 ceph-mon[32309]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:23.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:23 smithi073 ceph-mon[32309]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:23.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:23 smithi102 ceph-mon[36106]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:23.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:23 smithi102 ceph-mon[36106]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:23.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:23 smithi179 ceph-mon[36637]: from='client.15264 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:23.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:23 smithi179 ceph-mon[36637]: pgmap v362: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:24.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 -- 172.21.15.73:0/376597908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 msgr2=0x7f97341075b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:24.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 --2- 172.21.15.73:0/376597908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f97341075b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f9730009990 tx=0x7f9730009ca0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 -- 172.21.15.73:0/376597908 shutdown_connections 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 --2- 172.21.15.73:0/376597908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734107af0 0x7f9734109f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 --2- 172.21.15.73:0/376597908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734102890 0x7f9734104c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 --2- 172.21.15.73:0/376597908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f97341075b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 -- 172.21.15.73:0/376597908 >> 172.21.15.73:0/376597908 conn(0x7f97340fc2c0 msgr2=0x7f97340fe6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 -- 172.21.15.73:0/376597908 shutdown_connections 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.091+0000 7f973bfd2700 1 -- 172.21.15.73:0/376597908 wait complete. 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 Processor -- start 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 -- start start 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734102890 0x7f97341021a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:24.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734107af0 0x7f9734100d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973410da50 con 0x7f97341051c0 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973410d8d0 con 0x7f9734107af0 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973bfd2700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f973410dbd0 con 0x7f9734102890 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973956d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973956d700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49262/0 (socket says 172.21.15.73:49262) 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.092+0000 7f973956d700 1 -- 172.21.15.73:0/1196197958 learned_addr learned my addr 172.21.15.73:0/1196197958 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f9739d6e700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734102890 0x7f97341021a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973a56f700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734107af0 0x7f9734100d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 -- 172.21.15.73:0/1196197958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734102890 msgr2=0x7f97341021a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:24.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734102890 0x7f97341021a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 -- 172.21.15.73:0/1196197958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734107af0 msgr2=0x7f9734100d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734107af0 0x7f9734100d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 -- 172.21.15.73:0/1196197958 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9734101650 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973956d700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97300098e0 tx=0x7f97300057c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9730027070 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9730004150 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97341a2100 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f973000f460 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.093+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97341a2680 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.095+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9734068aa0 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.095+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f973002a070 con 0x7f97341051c0 2024-03-09T00:06:24.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.096+0000 7f972affd700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 0x7f97200a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:24.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.096+0000 7f9739d6e700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 0x7f97200a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:24.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.096+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f97300cfb90 con 0x7f97341051c0 2024-03-09T00:06:24.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.096+0000 7f9739d6e700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 0x7f97200a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f972400ac50 tx=0x7f972400a380 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:24.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.101+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9730013070 con 0x7f97341051c0 2024-03-09T00:06:24.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.347+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f973402dbe0 con 0x7f97341051c0 2024-03-09T00:06:24.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.347+0000 7f972affd700 1 -- 172.21.15.73:0/1196197958 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 20 v20) v1 ==== 76+0+2919 (secure 0 0 0) 0x7f973009c8f0 con 0x7f97341051c0 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:e20 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:legacy client fscid: 1 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:Filesystem 'cephfs' (1) 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:fs_name cephfs 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 20 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-03-09T00:04:13.661086+0000 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-03-09T00:04:39.321347+0000 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:tableserver 0 2024-03-09T00:06:24.343 INFO:teuthology.orchestra.run.smithi073.stdout:root 0 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:session_timeout 300 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:session_autoclose 300 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:max_file_size 1099511627776 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:max_xattr_size 65536 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:required_client_features {} 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure 0 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:last_failure_osd_epoch 0 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:max_mds 1 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:in 0 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:up {0=24515} 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:failed 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:damaged 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:stopped 2024-03-09T00:06:24.344 INFO:teuthology.orchestra.run.smithi073.stdout:data_pools [3,4] 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:metadata_pool 2 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:inline_data disabled 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:balancer 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:bal_rank_mask -1 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:standby_count_wanted 0 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.b{0:24515} state up:active seq 3 addr [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.i{0:24493} state up:standby-replay seq 1 addr [v2:172.21.15.179:6838/2754171892,v1:172.21.15.179:6839/2754171892] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:Standby daemons: 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.j{-1:14556} state up:standby seq 1 addr [v2:172.21.15.73:6834/1664042644,v1:172.21.15.73:6835/1664042644] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.d{-1:14568} state up:standby seq 1 addr [v2:172.21.15.73:6836/3028984722,v1:172.21.15.73:6837/3028984722] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.345 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.g{-1:14631} state up:standby seq 1 addr [v2:172.21.15.73:6838/2046390468,v1:172.21.15.73:6839/2046390468] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.a{-1:14649} state up:standby seq 1 addr [v2:172.21.15.73:6840/2662738077,v1:172.21.15.73:6841/2662738077] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.l{-1:24406} state up:standby seq 1 addr [v2:172.21.15.179:6832/3042063350,v1:172.21.15.179:6833/3042063350] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.k{-1:24410} state up:standby seq 1 addr [v2:172.21.15.102:6832/2292586216,v1:172.21.15.102:6833/2292586216] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.e{-1:24446} state up:standby seq 1 addr [v2:172.21.15.102:6834/364340358,v1:172.21.15.102:6835/364340358] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.f{-1:24457} state up:standby seq 1 addr [v2:172.21.15.179:6834/2774016402,v1:172.21.15.179:6835/2774016402] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.c{-1:24475} state up:standby seq 1 addr [v2:172.21.15.179:6836/2684774857,v1:172.21.15.179:6837/2684774857] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.346 INFO:teuthology.orchestra.run.smithi073.stdout:[mds.h{-1:24479} state up:standby seq 1 addr [v2:172.21.15.102:6836/306199757,v1:172.21.15.102:6837/306199757] compat {c=[1],r=[1],i=[fff]}] 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.349+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 msgr2=0x7f97200a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.349+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 0x7f97200a4dd0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f972400ac50 tx=0x7f972400a380 comp rx=0 tx=0).stop 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.349+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 msgr2=0x7f9734100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.349+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 crypto rx=0x7f97300098e0 tx=0x7f97300057c0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 shutdown_connections 2024-03-09T00:06:24.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f97200a2910 0x7f97200a4dd0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9734107af0 0x7f9734100d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9734102890 0x7f97341021a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 --2- 172.21.15.73:0/1196197958 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f97341051c0 0x7f9734100840 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 >> 172.21.15.73:0/1196197958 conn(0x7f97340fc2c0 msgr2=0x7f97341031d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 shutdown_connections 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:24.350+0000 7f973bfd2700 1 -- 172.21.15.73:0/1196197958 wait complete. 2024-03-09T00:06:24.348 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:25.018 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-09T00:06:25.266 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:25.325 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:25 smithi073 ceph-mon[32309]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:25.325 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:25 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1196197958' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-09T00:06:25.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:25 smithi102 ceph-mon[36106]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:25.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:25 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1196197958' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-09T00:06:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:25 smithi179 ceph-mon[36637]: pgmap v363: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:25 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1196197958' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-09T00:06:27.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 -- 172.21.15.73:0/3644541382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 msgr2=0x7f0534104c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 --2- 172.21.15.73:0/3644541382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f0534104c90 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f052c009990 tx=0x7f052c009ca0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 -- 172.21.15.73:0/3644541382 shutdown_connections 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 --2- 172.21.15.73:0/3644541382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f0534104c90 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 --2- 172.21.15.73:0/3644541382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0534107b00 0x7f0534109f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 --2- 172.21.15.73:0/3644541382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f05341051d0 0x7f05341075c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 -- 172.21.15.73:0/3644541382 >> 172.21.15.73:0/3644541382 conn(0x7f05340fc2b0 msgr2=0x7f05340fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:27.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 -- 172.21.15.73:0/3644541382 shutdown_connections 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.025+0000 7f053b451700 1 -- 172.21.15.73:0/3644541382 wait complete. 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 Processor -- start 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 -- start start 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 0x7f0534100850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0534107b00 0x7f0534100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053410da80 con 0x7f05341028a0 2024-03-09T00:06:27.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053410d900 con 0x7f05341051d0 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f053b451700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f053410dc00 con 0x7f0534107b00 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f05391ed700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f05391ed700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49288/0 (socket says 172.21.15.73:49288) 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f05391ed700 1 -- 172.21.15.73:0/2614874756 learned_addr learned my addr 172.21.15.73:0/2614874756 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f05399ee700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0534107b00 0x7f0534100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.026+0000 7f05389ec700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 0x7f0534100850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 -- 172.21.15.73:0/2614874756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0534107b00 msgr2=0x7f0534100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:27.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0534107b00 0x7f0534100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 -- 172.21.15.73:0/2614874756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 msgr2=0x7f0534100850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 0x7f0534100850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 -- 172.21.15.73:0/2614874756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0534101660 con 0x7f05341028a0 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05389ec700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 0x7f0534100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05391ed700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f052c0098e0 tx=0x7f052c005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c027070 con 0x7f05341028a0 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f052c0059c0 con 0x7f05341028a0 2024-03-09T00:06:27.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05341a2230 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f052c022e40 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.027+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05341a2760 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.028+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0534068aa0 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.029+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f052c02a070 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.029+0000 7f05327fc700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 0x7f051c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.029+0000 7f05389ec700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 0x7f051c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.030+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f052c0cfd20 con 0x7f05341028a0 2024-03-09T00:06:27.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.030+0000 7f05389ec700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 0x7f051c0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:27.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.033+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f052c013070 con 0x7f05341028a0 2024-03-09T00:06:27.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.231+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7f053402d830 con 0x7f05341028a0 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.232+0000 7f05327fc700 1 -- 172.21.15.73:0/2614874756 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v93) v1 ==== 56+0+5675 (secure 0 0 0) 0x7f052c09caa0 con 0x7f05341028a0 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 93 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:fsid 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-03-08T23:56:09.713229+0000 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:modified 2024-03-09T00:06:06.068347+0000 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-09T00:06:27.227 INFO:teuthology.orchestra.run.smithi073.stdout:crush_version 27 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:full_ratio 0.95 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:backfillfull_ratio 0.9 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:nearfull_ratio 0.85 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:require_min_compat_client luminous 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:min_compat_client luminous 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:require_osd_release squid 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:stretch_mode_enabled false 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 84 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.05 application cephfs read_balance_score 1.50 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:pool 4 'cephfs_data_ec' erasure profile cephfs_data_ec size 4 min_size 3 crush_rule 1 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 89 flags hashpspool,ec_overwrites stripe_width 8192 pg_num_min 64 target_size_ratio 0.9 application cephfs 2024-03-09T00:06:27.228 INFO:teuthology.orchestra.run.smithi073.stdout:max_osd 12 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6802/3305091634,v1:172.21.15.73:6803/3305091634] [v2:172.21.15.73:6804/3305091634,v1:172.21.15.73:6805/3305091634] exists,up 23a8d6f6-f2b8-449c-8948-2a7287b837a9 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6800/491071057,v1:172.21.15.102:6801/491071057] [v2:172.21.15.102:6802/491071057,v1:172.21.15.102:6803/491071057] exists,up c4e798b8-2a59-4ab9-8e1b-687cb0a3099d 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.2 up in weight 1 up_from 18 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6800/2018456550,v1:172.21.15.179:6801/2018456550] [v2:172.21.15.179:6802/2018456550,v1:172.21.15.179:6803/2018456550] exists,up ba842f27-768e-4c79-827f-4e0ca1411d87 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.3 up in weight 1 up_from 25 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6810/3007966557,v1:172.21.15.73:6811/3007966557] [v2:172.21.15.73:6812/3007966557,v1:172.21.15.73:6813/3007966557] exists,up b9390f11-f076-4591-b204-73187536054b 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.4 up in weight 1 up_from 33 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6808/1213471038,v1:172.21.15.102:6809/1213471038] [v2:172.21.15.102:6810/1213471038,v1:172.21.15.102:6811/1213471038] exists,up be6f4676-0213-47ac-806e-52fe6bddb9e2 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.5 up in weight 1 up_from 38 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6808/833646001,v1:172.21.15.179:6809/833646001] [v2:172.21.15.179:6810/833646001,v1:172.21.15.179:6811/833646001] exists,up 8b052756-28e2-438c-ad07-0e489a79f2f2 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.6 up in weight 1 up_from 46 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6818/1949163006,v1:172.21.15.73:6819/1949163006] [v2:172.21.15.73:6820/1949163006,v1:172.21.15.73:6821/1949163006] exists,up 4be6204e-f4a1-4605-a89f-3b3e222336c1 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.7 up in weight 1 up_from 51 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6816/1887369896,v1:172.21.15.102:6817/1887369896] [v2:172.21.15.102:6818/1887369896,v1:172.21.15.102:6819/1887369896] exists,up e08563fd-4504-49ad-9b28-33db12548e5e 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.8 up in weight 1 up_from 59 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6816/224478801,v1:172.21.15.179:6817/224478801] [v2:172.21.15.179:6818/224478801,v1:172.21.15.179:6819/224478801] exists,up 0b60fba7-28c7-455e-9120-944eaa7f08dd 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.9 up in weight 1 up_from 67 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.73:6826/1773882248,v1:172.21.15.73:6827/1773882248] [v2:172.21.15.73:6828/1773882248,v1:172.21.15.73:6829/1773882248] exists,up f6075430-0ebe-4194-9acd-b33cbed4bdd1 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.10 up in weight 1 up_from 72 up_thru 90 down_at 0 last_clean_interval [0,0) [v2:172.21.15.102:6824/2526045067,v1:172.21.15.102:6825/2526045067] [v2:172.21.15.102:6826/2526045067,v1:172.21.15.102:6827/2526045067] exists,up 9efc1e41-835a-496e-8487-b158cca2925a 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:osd.11 up in weight 1 up_from 80 up_thru 92 down_at 0 last_clean_interval [0,0) [v2:172.21.15.179:6824/897506355,v1:172.21.15.179:6825/897506355] [v2:172.21.15.179:6826/897506355,v1:172.21.15.179:6827/897506355] exists,up 48124e49-5bc5-49ff-b7b4-c139fcb1346f 2024-03-09T00:06:27.229 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 2.5 [8,11] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 2.10 [8,11] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 2.16 [2,4] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 2.1d [2,4] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 3.2 [3,9] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 3.e [5,9] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 3.13 [4,9] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 3.1b [4,6] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 3.3a [4,8] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 4.7 [5,8] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 4.10 [9,8] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 4.12 [9,10] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:pg_upmap_items 4.19 [9,8] 2024-03-09T00:06:27.230 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:6801/4041937587 expires 2024-03-09T23:57:03.831065+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:6800/4041937587 expires 2024-03-09T23:57:03.831065+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/3726094398 expires 2024-03-09T23:57:03.831065+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:6800/1574449933 expires 2024-03-09T23:56:36.296712+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/1410011990 expires 2024-03-09T23:56:36.296712+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/545419019 expires 2024-03-09T23:57:03.831065+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:6801/1574449933 expires 2024-03-09T23:56:36.296712+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/4143923877 expires 2024-03-09T23:56:36.296712+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/161900832 expires 2024-03-09T23:57:03.831065+0000 2024-03-09T00:06:27.231 INFO:teuthology.orchestra.run.smithi073.stdout:blocklist 172.21.15.73:0/383471347 expires 2024-03-09T23:56:36.296712+0000 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 msgr2=0x7f051c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 0x7f051c0a4d50 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f0528005fd0 tx=0x7f0528005ee0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 msgr2=0x7f05341021b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f052c0098e0 tx=0x7f052c005e80 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 shutdown_connections 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f05341028a0 0x7f05341021b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f051c0a2890 0x7f051c0a4d50 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0534107b00 0x7f0534100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 --2- 172.21.15.73:0/2614874756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f05341051d0 0x7f0534100850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:27.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 >> 172.21.15.73:0/2614874756 conn(0x7f05340fc2b0 msgr2=0x7f0534103140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:27.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 shutdown_connections 2024-03-09T00:06:27.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:27.234+0000 7f053b451700 1 -- 172.21.15.73:0/2614874756 wait complete. 2024-03-09T00:06:27.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:27 smithi073 ceph-mon[32309]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:27.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:27 smithi102 ceph-mon[36106]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:27.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:27 smithi179 ceph-mon[36637]: pgmap v364: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:27.955 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-09T00:06:27.975 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 -- 172.21.15.73:0/3669994591 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194101310 msgr2=0x7fc194101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 --2- 172.21.15.73:0/3669994591 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194101310 0x7fc194101730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fc184009900 tx=0x7fc18402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 -- 172.21.15.73:0/3669994591 shutdown_connections 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 --2- 172.21.15.73:0/3669994591 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194102ec0 0x7fc194109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 --2- 172.21.15.73:0/3669994591 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102500 0x7fc194102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 --2- 172.21.15.73:0/3669994591 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194101310 0x7fc194101730 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 -- 172.21.15.73:0/3669994591 >> 172.21.15.73:0/3669994591 conn(0x7fc1940fc8a0 msgr2=0x7fc1940fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 -- 172.21.15.73:0/3669994591 shutdown_connections 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.090+0000 7fc199412640 1 -- 172.21.15.73:0/3669994591 wait complete. 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 Processor -- start 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 -- start start 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 0x7fc194197700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 0x7fc19419ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19410d060 con 0x7fc194102500 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19410cee0 con 0x7fc194101310 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc199412640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc19410d1e0 con 0x7fc194102ec0 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49314/0 (socket says 172.21.15.73:49314) 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 -- 172.21.15.73:0/685735190 learned_addr learned my addr 172.21.15.73:0/685735190 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc192ffd640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 0x7fc194197700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 -- 172.21.15.73:0/685735190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 msgr2=0x7fc19419ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1937fe640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 0x7fc19419ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 0x7fc19419ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 -- 172.21.15.73:0/685735190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 msgr2=0x7fc194197700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 0x7fc194197700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 -- 172.21.15.73:0/685735190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc19419f320 con 0x7fc194102500 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc192ffd640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 0x7fc194197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1937fe640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 0x7fc19419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.091+0000 7fc1927fc640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b4a0 tx=0x7fc18800b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.092+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188014070 con 0x7fc194102500 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.092+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc19419f610 con 0x7fc194102500 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.092+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc188010af0 con 0x7fc194102500 2024-03-09T00:06:28.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.092+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc188017d10 con 0x7fc194102500 2024-03-09T00:06:28.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.092+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc19419fb60 con 0x7fc194102500 2024-03-09T00:06:28.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.093+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc194108630 con 0x7fc194102500 2024-03-09T00:06:28.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.094+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc188017070 con 0x7fc194102500 2024-03-09T00:06:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.094+0000 7fc17bfff640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 0x7fc17c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.094+0000 7fc192ffd640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 0x7fc17c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.094+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc1880ce6d0 con 0x7fc194102500 2024-03-09T00:06:28.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.095+0000 7fc192ffd640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 0x7fc17c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc184006010 tx=0x7fc184000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.098+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc18809b3c0 con 0x7fc194102500 2024-03-09T00:06:28.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.275+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc19402d090 con 0x7fc194102500 2024-03-09T00:06:28.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.277+0000 7fc17bfff640 1 -- 172.21.15.73:0/685735190 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7fc18809acc0 con 0x7fc194102500 2024-03-09T00:06:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:28.273 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:28.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 msgr2=0x7fc17c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 0x7fc17c0a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fc184006010 tx=0x7fc184000bc0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 msgr2=0x7fc194197c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fc18800b4a0 tx=0x7fc18800b960 comp rx=0 tx=0).stop 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 shutdown_connections 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc17c0a2800 0x7fc17c0a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc194102ec0 0x7fc19419ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc194102500 0x7fc194197c40 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 --2- 172.21.15.73:0/685735190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc194101310 0x7fc194197700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 >> 172.21.15.73:0/685735190 conn(0x7fc1940fc8a0 msgr2=0x7fc194105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 shutdown_connections 2024-03-09T00:06:28.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.279+0000 7fc199412640 1 -- 172.21.15.73:0/685735190 wait complete. 2024-03-09T00:06:28.284 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:28 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2614874756' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-09T00:06:28.284 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:28 smithi073 ceph-mon[32309]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:28.286 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:28.287 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:28.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 -- 172.21.15.73:0/321890297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5601013f0 msgr2=0x7fb560101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 --2- 172.21.15.73:0/321890297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5601013f0 0x7fb560101870 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7fb550009900 tx=0x7fb55002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:28.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 -- 172.21.15.73:0/321890297 shutdown_connections 2024-03-09T00:06:28.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 --2- 172.21.15.73:0/321890297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb560101db0 0x7fb560108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 --2- 172.21.15.73:0/321890297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5601013f0 0x7fb560101870 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 --2- 172.21.15.73:0/321890297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb560100200 0x7fb560100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 -- 172.21.15.73:0/321890297 >> 172.21.15.73:0/321890297 conn(0x7fb5600fb790 msgr2=0x7fb5600fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 -- 172.21.15.73:0/321890297 shutdown_connections 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 -- 172.21.15.73:0/321890297 wait complete. 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.398+0000 7fb566f22640 1 Processor -- start 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 -- start start 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5601013f0 0x7fb56019c310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb560101db0 0x7fb56019e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56010be90 con 0x7fb560100200 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56010bd10 con 0x7fb5601013f0 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb56010c010 con 0x7fb560101db0 2024-03-09T00:06:28.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49340/0 (socket says 172.21.15.73:49340) 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 -- 172.21.15.73:0/3813269617 learned_addr learned my addr 172.21.15.73:0/3813269617 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb557fff640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5601013f0 0x7fb56019c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 -- 172.21.15.73:0/3813269617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb560101db0 msgr2=0x7fb56019e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb560101db0 0x7fb56019e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 -- 172.21.15.73:0/3813269617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5601013f0 msgr2=0x7fb56019c310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5601013f0 0x7fb56019c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 -- 172.21.15.73:0/3813269617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56019f210 con 0x7fb560100200 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb564cbe640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00ccb0 tx=0x7fb55c00a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c003990 con 0x7fb560100200 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb55c003b30 con 0x7fb560100200 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb55c00f7b0 con 0x7fb560100200 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.399+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb56019f500 con 0x7fb560100200 2024-03-09T00:06:28.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.400+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb56019fa00 con 0x7fb560100200 2024-03-09T00:06:28.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.401+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5601075c0 con 0x7fb560100200 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.401+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fb55c00f950 con 0x7fb560100200 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.402+0000 7fb555ffb640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 0x7fb5480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.402+0000 7fb557fff640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 0x7fb5480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.402+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fb55c0cefc0 con 0x7fb560100200 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.402+0000 7fb557fff640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 0x7fb5480a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb550036040 tx=0x7fb550005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.406+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb55c014030 con 0x7fb560100200 2024-03-09T00:06:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:28 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2614874756' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-09T00:06:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:28 smithi102 ceph-mon[36106]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:28.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.580+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fb56002ce70 con 0x7fb560100200 2024-03-09T00:06:28.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.580+0000 7fb555ffb640 1 -- 172.21.15.73:0/3813269617 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7fb56002ce70 con 0x7fb560100200 2024-03-09T00:06:28.575 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 msgr2=0x7fb5480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 0x7fb5480a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fb550036040 tx=0x7fb550005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 msgr2=0x7fb56006a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7fb55c00ccb0 tx=0x7fb55c00a620 comp rx=0 tx=0).stop 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 shutdown_connections 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fb5480a2800 0x7fb5480a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb560101db0 0x7fb56019e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5601013f0 0x7fb56019c310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 --2- 172.21.15.73:0/3813269617 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb560100200 0x7fb56006a470 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 >> 172.21.15.73:0/3813269617 conn(0x7fb5600fb790 msgr2=0x7fb560104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 shutdown_connections 2024-03-09T00:06:28.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.583+0000 7fb566f22640 1 -- 172.21.15.73:0/3813269617 wait complete. 2024-03-09T00:06:28.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:28 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2614874756' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-09T00:06:28.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:28 smithi179 ceph-mon[36637]: pgmap v365: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:28.589 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:28.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.698+0000 7f613c559640 1 -- 172.21.15.73:0/1977417054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6134101300 msgr2=0x7f6134101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.698+0000 7f613c559640 1 --2- 172.21.15.73:0/1977417054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6134101300 0x7f6134101720 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f6124009900 tx=0x7f612402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- 172.21.15.73:0/1977417054 shutdown_connections 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- 172.21.15.73:0/1977417054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6134102eb0 0x7f6134109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- 172.21.15.73:0/1977417054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61341024f0 0x7f6134102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- 172.21.15.73:0/1977417054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6134101300 0x7f6134101720 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- 172.21.15.73:0/1977417054 >> 172.21.15.73:0/1977417054 conn(0x7f61340fc890 msgr2=0x7f61340fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- 172.21.15.73:0/1977417054 shutdown_connections 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- 172.21.15.73:0/1977417054 wait complete. 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 Processor -- start 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- start start 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 0x7f61340750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6134102eb0 0x7f6134073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613410cf90 con 0x7f61341024f0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613410ce10 con 0x7f6134102eb0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f613c559640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f613410d110 con 0x7f6134101300 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f6139af4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f6139af4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49128/0 (socket says 172.21.15.73:49128) 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.699+0000 7f6139af4640 1 -- 172.21.15.73:0/452798295 learned_addr learned my addr 172.21.15.73:0/452798295 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f613a2f5640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 0x7f61340750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 -- 172.21.15.73:0/452798295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 msgr2=0x7f61340750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 0x7f61340750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 -- 172.21.15.73:0/452798295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6134102eb0 msgr2=0x7f6134073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6134102eb0 0x7f6134073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 -- 172.21.15.73:0/452798295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61340744b0 con 0x7f61341024f0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f613a2f5640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 0x7f61340750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f6139af4640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f613000d770 tx=0x7f613000dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130017070 con 0x7f61341024f0 2024-03-09T00:06:28.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61341a17e0 con 0x7f61341024f0 2024-03-09T00:06:28.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6130005500 con 0x7f61341024f0 2024-03-09T00:06:28.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130004e00 con 0x7f61341024f0 2024-03-09T00:06:28.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.700+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61341a1ce0 con 0x7f61341024f0 2024-03-09T00:06:28.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.701+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61341086c0 con 0x7f61341024f0 2024-03-09T00:06:28.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.702+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6130004fa0 con 0x7f61341024f0 2024-03-09T00:06:28.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.703+0000 7f612b7fe640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 0x7f61200a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.703+0000 7f613a2f5640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 0x7f61200a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.703+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f61300ced40 con 0x7f61341024f0 2024-03-09T00:06:28.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.703+0000 7f613a2f5640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 0x7f61200a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f612402c7b0 tx=0x7f6124005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.706+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f613009ba30 con 0x7f61341024f0 2024-03-09T00:06:28.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.879+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f613402d070 con 0x7f61341024f0 2024-03-09T00:06:28.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.879+0000 7f612b7fe640 1 -- 172.21.15.73:0/452798295 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f613009b330 con 0x7f61341024f0 2024-03-09T00:06:28.874 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 msgr2=0x7f61200a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 0x7f61200a4cc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f612402c7b0 tx=0x7f6124005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 msgr2=0x7f61340736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f613000d770 tx=0x7f613000dc30 comp rx=0 tx=0).stop 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 shutdown_connections 2024-03-09T00:06:28.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f61200a2800 0x7f61200a4cc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6134102eb0 0x7f6134073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61341024f0 0x7f61340736e0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 --2- 172.21.15.73:0/452798295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6134101300 0x7f61340750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 >> 172.21.15.73:0/452798295 conn(0x7f61340fc890 msgr2=0x7f6134105890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 shutdown_connections 2024-03-09T00:06:28.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.881+0000 7f613c559640 1 -- 172.21.15.73:0/452798295 wait complete. 2024-03-09T00:06:28.887 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:28.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 -- 172.21.15.73:0/1844710538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828102eb0 msgr2=0x7fe828109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 --2- 172.21.15.73:0/1844710538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828102eb0 0x7fe828109740 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7fe820009ca0 tx=0x7fe82002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 -- 172.21.15.73:0/1844710538 shutdown_connections 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 --2- 172.21.15.73:0/1844710538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828102eb0 0x7fe828109740 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 --2- 172.21.15.73:0/1844710538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 0x7fe828102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 --2- 172.21.15.73:0/1844710538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828101300 0x7fe828101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 -- 172.21.15.73:0/1844710538 >> 172.21.15.73:0/1844710538 conn(0x7fe8280fc890 msgr2=0x7fe8280fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 -- 172.21.15.73:0/1844710538 shutdown_connections 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.997+0000 7fe82de59640 1 -- 172.21.15.73:0/1844710538 wait complete. 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 Processor -- start 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 -- start start 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 0x7fe828197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 0x7fe82819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82810cf90 con 0x7fe828101300 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82810ce10 con 0x7fe828102eb0 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe82de59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe82810d110 con 0x7fe8281024f0 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe8277fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe8277fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49138/0 (socket says 172.21.15.73:49138) 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.998+0000 7fe8277fe640 1 -- 172.21.15.73:0/3643496114 learned_addr learned my addr 172.21.15.73:0/3643496114 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe826ffd640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 0x7fe828197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 -- 172.21.15.73:0/3643496114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 msgr2=0x7fe828197b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 0x7fe828197b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 -- 172.21.15.73:0/3643496114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 msgr2=0x7fe82819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe827fff640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 0x7fe82819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 0x7fe82819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:28.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 -- 172.21.15.73:0/3643496114 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe82819f1e0 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe827fff640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 0x7fe82819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe8277fe640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe81800d950 tx=0x7fe81800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe818017020 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe818009c30 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe82819f4d0 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe81800f720 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:28.999+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe82819fa70 con 0x7fe828101300 2024-03-09T00:06:28.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.000+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe828108620 con 0x7fe828101300 2024-03-09T00:06:28.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.001+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe81801a020 con 0x7fe828101300 2024-03-09T00:06:28.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.002+0000 7fe824ff9640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 0x7fe8100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:28.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.002+0000 7fe826ffd640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 0x7fe8100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:28.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.002+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe8180ced70 con 0x7fe828101300 2024-03-09T00:06:28.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.002+0000 7fe826ffd640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 0x7fe8100a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005fd0 tx=0x7fe81c00b510 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.006+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe818021020 con 0x7fe828101300 2024-03-09T00:06:29.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/685735190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:29.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3813269617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:29 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/452798295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.181+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe828061f20 con 0x7fe828101300 2024-03-09T00:06:29.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.181+0000 7fe824ff9640 1 -- 172.21.15.73:0/3643496114 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe818009de0 con 0x7fe828101300 2024-03-09T00:06:29.176 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 msgr2=0x7fe8100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 0x7fe8100a4d40 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fe81c005fd0 tx=0x7fe81c00b510 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 msgr2=0x7fe8281975c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7fe81800d950 tx=0x7fe81800de10 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 shutdown_connections 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe8100a2880 0x7fe8100a4d40 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe828102eb0 0x7fe82819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe8281024f0 0x7fe828197b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 --2- 172.21.15.73:0/3643496114 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe828101300 0x7fe8281975c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 >> 172.21.15.73:0/3643496114 conn(0x7fe8280fc890 msgr2=0x7fe8281057f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 shutdown_connections 2024-03-09T00:06:29.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.183+0000 7fe82de59640 1 -- 172.21.15.73:0/3643496114 wait complete. 2024-03-09T00:06:29.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:29.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 -- 172.21.15.73:0/3551416896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc630100e90 msgr2=0x7fc630108730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 --2- 172.21.15.73:0/3551416896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc630100e90 0x7fc630108730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7fc62c009ca0 tx=0x7fc62c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 -- 172.21.15.73:0/3551416896 shutdown_connections 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 --2- 172.21.15.73:0/3551416896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc630100e90 0x7fc630108730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 --2- 172.21.15.73:0/3551416896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 0x7fc630100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 --2- 172.21.15.73:0/3551416896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc6301017a0 0x7fc630101bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.298+0000 7fc63806f640 1 -- 172.21.15.73:0/3551416896 >> 172.21.15.73:0/3551416896 conn(0x7fc6300faee0 msgr2=0x7fc6300fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- 172.21.15.73:0/3551416896 shutdown_connections 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- 172.21.15.73:0/3551416896 wait complete. 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 Processor -- start 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- start start 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 0x7fc6300fff90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc630100e90 0x7fc63019c270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63010beb0 con 0x7fc6301017a0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63010bd30 con 0x7fc630100e90 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63806f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63010c030 con 0x7fc6301004d0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63660c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63660c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49154/0 (socket says 172.21.15.73:49154) 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.299+0000 7fc63660c640 1 -- 172.21.15.73:0/3975959516 learned_addr learned my addr 172.21.15.73:0/3975959516 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc635e0b640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 0x7fc6300fff90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63560a640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc630100e90 0x7fc63019c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 -- 172.21.15.73:0/3975959516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 msgr2=0x7fc6300fff90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 0x7fc6300fff90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 -- 172.21.15.73:0/3975959516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc630100e90 msgr2=0x7fc63019c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc630100e90 0x7fc63019c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 -- 172.21.15.73:0/3975959516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc63019f1a0 con 0x7fc6301017a0 2024-03-09T00:06:29.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63660c640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02c700 tx=0x7fc62c0346f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c041020 con 0x7fc6301017a0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63019f430 con 0x7fc6301017a0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc62c032890 con 0x7fc6301017a0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62c039040 con 0x7fc6301017a0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.300+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc63019f970 con 0x7fc6301017a0 2024-03-09T00:06:29.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.301+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6301075b0 con 0x7fc6301017a0 2024-03-09T00:06:29.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.302+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc62c044020 con 0x7fc6301017a0 2024-03-09T00:06:29.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.303+0000 7fc626ffd640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 0x7fc61c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.303+0000 7fc635e0b640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 0x7fc61c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.303+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc62c0f0e30 con 0x7fc6301017a0 2024-03-09T00:06:29.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.303+0000 7fc635e0b640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 0x7fc61c0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc630069640 tx=0x7fc62800b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.307+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc62c04b020 con 0x7fc6301017a0 2024-03-09T00:06:29.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.481+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fc630061ef0 con 0x7fc6301017a0 2024-03-09T00:06:29.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.481+0000 7fc626ffd640 1 -- 172.21.15.73:0/3975959516 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fc62c0bdb20 con 0x7fc6301017a0 2024-03-09T00:06:29.476 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:29.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 msgr2=0x7fc61c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 0x7fc61c0a4cc0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fc630069640 tx=0x7fc62800b390 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 msgr2=0x7fc63019e8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fc62c02c700 tx=0x7fc62c0346f0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 shutdown_connections 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc61c0a2800 0x7fc61c0a4cc0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc6301017a0 0x7fc63019e8e0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc630100e90 0x7fc63019c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 --2- 172.21.15.73:0/3975959516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc6301004d0 0x7fc6300fff90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.483+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 >> 172.21.15.73:0/3975959516 conn(0x7fc6300faee0 msgr2=0x7fc630104780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.484+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 shutdown_connections 2024-03-09T00:06:29.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.484+0000 7fc63806f640 1 -- 172.21.15.73:0/3975959516 wait complete. 2024-03-09T00:06:29.489 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:29.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/685735190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:29.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3813269617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:29 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/452798295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/685735190' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:29.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3813269617' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:29 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/452798295' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:29.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 -- 172.21.15.73:0/1517739206 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 msgr2=0x7f12dc100960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 --2- 172.21.15.73:0/1517739206 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc100960 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7f12d0009930 tx=0x7f12d002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:29.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 -- 172.21.15.73:0/1517739206 shutdown_connections 2024-03-09T00:06:29.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 --2- 172.21.15.73:0/1517739206 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 0x7f12dc108740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 --2- 172.21.15.73:0/1517739206 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc100960 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 --2- 172.21.15.73:0/1517739206 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 0x7f12dc101c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 -- 172.21.15.73:0/1517739206 >> 172.21.15.73:0/1517739206 conn(0x7f12dc0faf30 msgr2=0x7f12dc0fd370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 -- 172.21.15.73:0/1517739206 shutdown_connections 2024-03-09T00:06:29.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.596+0000 7f12e21c4640 1 -- 172.21.15.73:0/1517739206 wait complete. 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 Processor -- start 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 -- start start 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 0x7f12dc197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 0x7f12dc19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10bf10 con 0x7f12dc1004e0 2024-03-09T00:06:29.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10bd90 con 0x7f12dc101840 2024-03-09T00:06:29.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12e21c4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12dc10c090 con 0x7f12dc100ea0 2024-03-09T00:06:29.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49168/0 (socket says 172.21.15.73:49168) 2024-03-09T00:06:29.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 -- 172.21.15.73:0/3079529969 learned_addr learned my addr 172.21.15.73:0/3079529969 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12daffd640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 0x7f12dc197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12dbfff640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 0x7f12dc19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 -- 172.21.15.73:0/3079529969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 msgr2=0x7f12dc197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 0x7f12dc197b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 -- 172.21.15.73:0/3079529969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 msgr2=0x7f12dc19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 0x7f12dc19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.597+0000 7f12db7fe640 1 -- 172.21.15.73:0/3079529969 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12dc19f260 con 0x7f12dc1004e0 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12db7fe640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00d770 tx=0x7f12cc00dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc017070 con 0x7f12dc1004e0 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12cc009c30 con 0x7f12dc1004e0 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12dc19f550 con 0x7f12dc1004e0 2024-03-09T00:06:29.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12cc003dd0 con 0x7f12dc1004e0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.598+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12dc19fa70 con 0x7f12dc1004e0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.599+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12dc19fcd0 con 0x7f12dc1004e0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.600+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f12cc003fb0 con 0x7f12dc1004e0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.601+0000 7f12d8ff9640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 0x7f12c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.601+0000 7f12daffd640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 0x7f12c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.601+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f12cc0ce6e0 con 0x7f12dc1004e0 2024-03-09T00:06:29.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.601+0000 7f12daffd640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 0x7f12c40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f12d0000c00 tx=0x7f12d0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.605+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f12cc009de0 con 0x7f12dc1004e0 2024-03-09T00:06:29.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.816+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f12dc1a0440 con 0x7f12dc1004e0 2024-03-09T00:06:29.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.817+0000 7f12d8ff9640 1 -- 172.21.15.73:0/3079529969 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f12cc09ad80 con 0x7f12dc1004e0 2024-03-09T00:06:29.812 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:29.812 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:29.812 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 msgr2=0x7f12c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 0x7f12c40a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f12d0000c00 tx=0x7f12d0005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 msgr2=0x7f12dc197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f12cc00d770 tx=0x7f12cc00dc30 comp rx=0 tx=0).stop 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 shutdown_connections 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f12c40a2800 0x7f12c40a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12dc101840 0x7f12dc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12dc100ea0 0x7f12dc197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 --2- 172.21.15.73:0/3079529969 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12dc1004e0 0x7f12dc197640 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 >> 172.21.15.73:0/3079529969 conn(0x7f12dc0faf30 msgr2=0x7f12dc0fc9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 shutdown_connections 2024-03-09T00:06:29.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.819+0000 7f12e21c4640 1 -- 172.21.15.73:0/3079529969 wait complete. 2024-03-09T00:06:29.826 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:29.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 -- 172.21.15.73:0/3173651908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa444104720 msgr2=0x7fa444104ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 --2- 172.21.15.73:0/3173651908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa444104720 0x7fa444104ba0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa438009900 tx=0x7fa43802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:29.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 -- 172.21.15.73:0/3173651908 shutdown_connections 2024-03-09T00:06:29.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 --2- 172.21.15.73:0/3173651908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa4441050e0 0x7fa44410b970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 --2- 172.21.15.73:0/3173651908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa444104720 0x7fa444104ba0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 --2- 172.21.15.73:0/3173651908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 0x7fa444103950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 -- 172.21.15.73:0/3173651908 >> 172.21.15.73:0/3173651908 conn(0x7fa4440feaa0 msgr2=0x7fa444100ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:29.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 -- 172.21.15.73:0/3173651908 shutdown_connections 2024-03-09T00:06:29.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.935+0000 7fa449ac0640 1 -- 172.21.15.73:0/3173651908 wait complete. 2024-03-09T00:06:29.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 Processor -- start 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 -- start start 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 0x7fa444199850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa444104720 0x7fa444199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44410f1c0 con 0x7fa4441050e0 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44410f040 con 0x7fa444103530 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa449ac0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa44410f340 con 0x7fa444104720 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa443fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa443fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49198/0 (socket says 172.21.15.73:49198) 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa442ffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa444104720 0x7fa444199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa443fff640 1 -- 172.21.15.73:0/1866373286 learned_addr learned my addr 172.21.15.73:0/1866373286 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.936+0000 7fa4437fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 0x7fa444199850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 -- 172.21.15.73:0/1866373286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa444104720 msgr2=0x7fa444199d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa444104720 0x7fa444199d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 -- 172.21.15.73:0/1866373286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 msgr2=0x7fa444199850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 0x7fa444199850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 -- 172.21.15.73:0/1866373286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4441a1470 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa443fff640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fd0 tx=0x7fa43c00b260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c017070 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4441a1760 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa43c003e50 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa43c0106c0 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.937+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4441a1c30 con 0x7fa4441050e0 2024-03-09T00:06:29.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.938+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa44410a8a0 con 0x7fa4441050e0 2024-03-09T00:06:29.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.939+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa43c003970 con 0x7fa4441050e0 2024-03-09T00:06:29.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.940+0000 7fa440ff9640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 0x7fa42c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:29.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.940+0000 7fa4437fe640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 0x7fa42c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:29.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.940+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa43c0ceb10 con 0x7fa4441050e0 2024-03-09T00:06:29.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.940+0000 7fa4437fe640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 0x7fa42c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa444104580 tx=0x7fa434005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:29.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:29.944+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa43c09b1b0 con 0x7fa4441050e0 2024-03-09T00:06:30.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.121+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa444061ef0 con 0x7fa4441050e0 2024-03-09T00:06:30.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.123+0000 7fa440ff9640 1 -- 172.21.15.73:0/1866373286 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7fa43c09f250 con 0x7fa4441050e0 2024-03-09T00:06:30.118 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:30.118 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 msgr2=0x7fa42c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 0x7fa42c0a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fa444104580 tx=0x7fa434005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 msgr2=0x7fa4441a0e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa43c009fd0 tx=0x7fa43c00b260 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 shutdown_connections 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa42c0a2800 0x7fa42c0a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa4441050e0 0x7fa4441a0e00 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa444104720 0x7fa444199d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 --2- 172.21.15.73:0/1866373286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa444103530 0x7fa444199850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 >> 172.21.15.73:0/1866373286 conn(0x7fa4440feaa0 msgr2=0x7fa444107a70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 shutdown_connections 2024-03-09T00:06:30.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.125+0000 7fa449ac0640 1 -- 172.21.15.73:0/1866373286 wait complete. 2024-03-09T00:06:30.132 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:30.132 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 -- 172.21.15.73:0/7879080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4102eb0 msgr2=0x7f15e4109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 --2- 172.21.15.73:0/7879080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4102eb0 0x7f15e4109740 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 crypto rx=0x7f15dc009ca0 tx=0x7f15dc02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 -- 172.21.15.73:0/7879080 shutdown_connections 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 --2- 172.21.15.73:0/7879080 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4102eb0 0x7f15e4109740 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 --2- 172.21.15.73:0/7879080 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e4102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 --2- 172.21.15.73:0/7879080 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4101300 0x7f15e4101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 -- 172.21.15.73:0/7879080 >> 172.21.15.73:0/7879080 conn(0x7f15e40fc890 msgr2=0x7f15e40fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 -- 172.21.15.73:0/7879080 shutdown_connections 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.242+0000 7f15ead7e640 1 -- 172.21.15.73:0/7879080 wait complete. 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 Processor -- start 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 -- start start 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e40736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 0x7f15e4073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e410cf60 con 0x7f15e4101300 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e410cde0 con 0x7f15e41024f0 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15ead7e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e410d0e0 con 0x7f15e4102eb0 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49212/0 (socket says 172.21.15.73:49212) 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 -- 172.21.15.73:0/2632701611 learned_addr learned my addr 172.21.15.73:0/2632701611 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e3fff640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e40736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e931b640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 0x7f15e4073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 -- 172.21.15.73:0/2632701611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 msgr2=0x7f15e4073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 0x7f15e4073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 -- 172.21.15.73:0/2632701611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 msgr2=0x7f15e40736e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e40736e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 -- 172.21.15.73:0/2632701611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e40744b0 con 0x7f15e4101300 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e931b640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 0x7f15e4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e3fff640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:30.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e8b1a640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f15d800e9c0 tx=0x7f15d800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.243+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d800cd80 con 0x7f15e4101300 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.244+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e41a16c0 con 0x7f15e4101300 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.244+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d80035f0 con 0x7f15e4101300 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.244+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8010690 con 0x7f15e4101300 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.244+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e41a1c10 con 0x7f15e4101300 2024-03-09T00:06:30.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.245+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15e41086a0 con 0x7f15e4101300 2024-03-09T00:06:30.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.246+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f15d80037a0 con 0x7f15e4101300 2024-03-09T00:06:30.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.246+0000 7f15e1ffb640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 0x7f15d40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.246+0000 7f15e3fff640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 0x7f15d40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.246+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f15d80cf3a0 con 0x7f15e4101300 2024-03-09T00:06:30.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.247+0000 7f15e3fff640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 0x7f15d40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15e4074e10 tx=0x7f15d0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.250+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f15d809c090 con 0x7f15e4101300 2024-03-09T00:06:30.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3643496114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3975959516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:30 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3079529969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:30.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:30 smithi073 ceph-mon[32309]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:30.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.427+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f15e402ce70 con 0x7f15e4101300 2024-03-09T00:06:30.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.428+0000 7f15e1ffb640 1 -- 172.21.15.73:0/2632701611 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f15e402ce70 con 0x7f15e4101300 2024-03-09T00:06:30.423 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 msgr2=0x7f15d40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 0x7f15d40a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f15e4074e10 tx=0x7f15d0005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 msgr2=0x7f15e40750a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 crypto rx=0x7f15d800e9c0 tx=0x7f15d800ee80 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 shutdown_connections 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f15d40a2800 0x7f15d40a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e4102eb0 0x7f15e4073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e41024f0 0x7f15e40736e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 --2- 172.21.15.73:0/2632701611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e4101300 0x7f15e40750a0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 >> 172.21.15.73:0/2632701611 conn(0x7f15e40fc890 msgr2=0x7f15e4105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 shutdown_connections 2024-03-09T00:06:30.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.430+0000 7f15ead7e640 1 -- 172.21.15.73:0/2632701611 wait complete. 2024-03-09T00:06:30.436 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:30.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3643496114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3975959516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:30 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3079529969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:30.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:30 smithi102 ceph-mon[36106]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:30.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 -- 172.21.15.73:0/565838262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0100210 msgr2=0x7fe1f0100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 --2- 172.21.15.73:0/565838262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0100210 0x7fe1f0100630 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec009900 tx=0x7fe1ec02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 -- 172.21.15.73:0/565838262 shutdown_connections 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 --2- 172.21.15.73:0/565838262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101dc0 0x7fe1f0108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 --2- 172.21.15.73:0/565838262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0101400 0x7fe1f0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 --2- 172.21.15.73:0/565838262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0100210 0x7fe1f0100630 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 -- 172.21.15.73:0/565838262 >> 172.21.15.73:0/565838262 conn(0x7fe1f00fb7a0 msgr2=0x7fe1f00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 -- 172.21.15.73:0/565838262 shutdown_connections 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.546+0000 7fe1f7673640 1 -- 172.21.15.73:0/565838262 wait complete. 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 Processor -- start 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 -- start start 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0100210 0x7fe1f006a450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101400 0x7fe1f019e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f010bea0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f010bd20 con 0x7fe1f0100210 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f7673640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1f010c020 con 0x7fe1f0101400 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f540f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0100210 0x7fe1f006a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49238/0 (socket says 172.21.15.73:49238) 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 -- 172.21.15.73:0/1053901261 learned_addr learned my addr 172.21.15.73:0/1053901261 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f4c0e640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101400 0x7fe1f019e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 -- 172.21.15.73:0/1053901261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101400 msgr2=0x7fe1f019e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101400 0x7fe1f019e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 -- 172.21.15.73:0/1053901261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0100210 msgr2=0x7fe1f006a450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.545 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.547+0000 7fe1f5c10640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0100210 0x7fe1f006a450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1f5c10640 1 -- 172.21.15.73:0/1053901261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1f01a14d0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1f5c10640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800eb10 tx=0x7fe1e800c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8004b20 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe1e8004cc0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1f01a1760 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe1e8010760 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.548+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1f01a1cb0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.549+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1f0107530 con 0x7fe1f0101dc0 2024-03-09T00:06:30.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.550+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe1e8005020 con 0x7fe1f0101dc0 2024-03-09T00:06:30.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.551+0000 7fe1e67fc640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 0x7fe1dc0a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.551+0000 7fe1f540f640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 0x7fe1dc0a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.551+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe1e80cf300 con 0x7fe1f0101dc0 2024-03-09T00:06:30.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.551+0000 7fe1f540f640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 0x7fe1dc0a4cf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.555+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe1e809bff0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3643496114' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3975959516' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:30 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3079529969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:30 smithi179 ceph-mon[36637]: pgmap v366: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.726+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fe1f0061ef0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.727+0000 7fe1e67fc640 1 -- 172.21.15.73:0/1053901261 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7fe1e809b8f0 con 0x7fe1f0101dc0 2024-03-09T00:06:30.722 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 msgr2=0x7fe1dc0a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 0x7fe1dc0a4cf0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fe1ec000c00 tx=0x7fe1ec005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 msgr2=0x7fe1f01a0ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fe1e800eb10 tx=0x7fe1e800c2d0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 shutdown_connections 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe1dc0a2830 0x7fe1dc0a4cf0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe1f0101dc0 0x7fe1f01a0ce0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe1f0101400 0x7fe1f019e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 --2- 172.21.15.73:0/1053901261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe1f0100210 0x7fe1f006a450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 >> 172.21.15.73:0/1053901261 conn(0x7fe1f00fb7a0 msgr2=0x7fe1f0104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 shutdown_connections 2024-03-09T00:06:30.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.729+0000 7fe1f7673640 1 -- 172.21.15.73:0/1053901261 wait complete. 2024-03-09T00:06:30.735 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:30.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 -- 172.21.15.73:0/1133079055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe540102eb0 msgr2=0x7fe540109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 --2- 172.21.15.73:0/1133079055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe540102eb0 0x7fe540109740 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7fe538009ca0 tx=0x7fe53802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 -- 172.21.15.73:0/1133079055 shutdown_connections 2024-03-09T00:06:30.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 --2- 172.21.15.73:0/1133079055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe540102eb0 0x7fe540109740 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 --2- 172.21.15.73:0/1133079055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe5401024f0 0x7fe540102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 --2- 172.21.15.73:0/1133079055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 0x7fe540101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.844+0000 7fe547e1f640 1 -- 172.21.15.73:0/1133079055 >> 172.21.15.73:0/1133079055 conn(0x7fe5400fc890 msgr2=0x7fe5400fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- 172.21.15.73:0/1133079055 shutdown_connections 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- 172.21.15.73:0/1133079055 wait complete. 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 Processor -- start 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- start start 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 0x7fe540197530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe540102eb0 0x7fe54019eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54010cf90 con 0x7fe5401024f0 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54010ce10 con 0x7fe540101300 2024-03-09T00:06:30.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe547e1f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe54010d110 con 0x7fe540102eb0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe5453ba640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe5453ba640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49254/0 (socket says 172.21.15.73:49254) 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe5453ba640 1 -- 172.21.15.73:0/652061149 learned_addr learned my addr 172.21.15.73:0/652061149 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe5463bc640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe540102eb0 0x7fe54019eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.845+0000 7fe545bbb640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 0x7fe540197530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 -- 172.21.15.73:0/652061149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe540102eb0 msgr2=0x7fe54019eae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe540102eb0 0x7fe54019eae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 -- 172.21.15.73:0/652061149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 msgr2=0x7fe540197530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 0x7fe540197530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 -- 172.21.15.73:0/652061149 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe54019f150 con 0x7fe5401024f0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe5453ba640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe53000c920 tx=0x7fe53000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530007c70 con 0x7fe5401024f0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe530007e10 con 0x7fe5401024f0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe54019f440 con 0x7fe5401024f0 2024-03-09T00:06:30.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe530018890 con 0x7fe5401024f0 2024-03-09T00:06:30.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.846+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe54019f990 con 0x7fe5401024f0 2024-03-09T00:06:30.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.847+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5401085c0 con 0x7fe5401024f0 2024-03-09T00:06:30.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.848+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe530018a70 con 0x7fe5401024f0 2024-03-09T00:06:30.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.849+0000 7fe536ffd640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 0x7fe52c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:30.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.849+0000 7fe545bbb640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 0x7fe52c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:30.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.849+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe5300cef10 con 0x7fe5401024f0 2024-03-09T00:06:30.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.849+0000 7fe545bbb640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 0x7fe52c0a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe540102350 tx=0x7fe53c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:30.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:30.853+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe530014030 con 0x7fe5401024f0 2024-03-09T00:06:31.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.026+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fe540061f20 con 0x7fe5401024f0 2024-03-09T00:06:31.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.027+0000 7fe536ffd640 1 -- 172.21.15.73:0/652061149 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7fe53009bc00 con 0x7fe5401024f0 2024-03-09T00:06:31.022 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:31.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 msgr2=0x7fe52c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 0x7fe52c0a4cc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fe540102350 tx=0x7fe53c00b390 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 msgr2=0x7fe540197a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7fe53000c920 tx=0x7fe53000cde0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 shutdown_connections 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe52c0a2800 0x7fe52c0a4cc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe540102eb0 0x7fe54019eae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe5401024f0 0x7fe540197a70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 --2- 172.21.15.73:0/652061149 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe540101300 0x7fe540197530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 >> 172.21.15.73:0/652061149 conn(0x7fe5400fc890 msgr2=0x7fe540105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 shutdown_connections 2024-03-09T00:06:31.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.029+0000 7fe547e1f640 1 -- 172.21.15.73:0/652061149 wait complete. 2024-03-09T00:06:31.035 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:31.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 -- 172.21.15.73:0/2533746166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102ec0 msgr2=0x7f5704109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 --2- 172.21.15.73:0/2533746166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102ec0 0x7f5704109750 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f56fc009ca0 tx=0x7f56fc02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 -- 172.21.15.73:0/2533746166 shutdown_connections 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 --2- 172.21.15.73:0/2533746166 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102ec0 0x7f5704109750 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 --2- 172.21.15.73:0/2533746166 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102500 0x7f5704102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 --2- 172.21.15.73:0/2533746166 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 0x7f5704101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 -- 172.21.15.73:0/2533746166 >> 172.21.15.73:0/2533746166 conn(0x7f57040fc880 msgr2=0x7f57040fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 -- 172.21.15.73:0/2533746166 shutdown_connections 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.147+0000 7f570a4bf640 1 -- 172.21.15.73:0/2533746166 wait complete. 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 Processor -- start 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 -- start start 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 0x7f5704197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 0x7f570419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570410cfa0 con 0x7f5704102500 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570410ce20 con 0x7f5704102ec0 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f570a4bf640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f570410d120 con 0x7f5704101310 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f57037fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f57037fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49278/0 (socket says 172.21.15.73:49278) 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f57037fe640 1 -- 172.21.15.73:0/1834045918 learned_addr learned my addr 172.21.15.73:0/1834045918 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.148+0000 7f5703fff640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 0x7f5704197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f5708a5c640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 0x7f570419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 -- 172.21.15.73:0/1834045918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 msgr2=0x7f5704197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 0x7f5704197620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 -- 172.21.15.73:0/1834045918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 msgr2=0x7f570419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 0x7f570419ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 -- 172.21.15.73:0/1834045918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f570419f240 con 0x7f5704102500 2024-03-09T00:06:31.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f5708a5c640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 0x7f570419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57037fe640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c890 tx=0x7f56f800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8004220 con 0x7f5704102500 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f570419f530 con 0x7f5704102500 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56f80043c0 con 0x7f5704102500 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56f8004a00 con 0x7f5704102500 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.149+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f570419fa80 con 0x7f5704102500 2024-03-09T00:06:31.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.150+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57041085d0 con 0x7f5704102500 2024-03-09T00:06:31.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.151+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f56f80079b0 con 0x7f5704102500 2024-03-09T00:06:31.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.151+0000 7f57017fa640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.151+0000 7f5703fff640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.152+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f56f80ceaf0 con 0x7f5704102500 2024-03-09T00:06:31.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.152+0000 7f5703fff640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f56f40058c0 tx=0x7f56f400a2d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.156+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f56f8014030 con 0x7f5704102500 2024-03-09T00:06:31.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.333+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f5704061f20 con 0x7f5704102500 2024-03-09T00:06:31.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.334+0000 7f57017fa640 1 -- 172.21.15.73:0/1834045918 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f56f809b7e0 con 0x7f5704102500 2024-03-09T00:06:31.329 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 msgr2=0x7f56ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f56f40058c0 tx=0x7f56f400a2d0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 msgr2=0x7f5704197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f56f800c890 tx=0x7f56f800cd50 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 shutdown_connections 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f56ec0a2800 0x7f56ec0a4cc0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5704102ec0 0x7f570419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5704102500 0x7f5704197b60 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 --2- 172.21.15.73:0/1834045918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5704101310 0x7f5704197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 >> 172.21.15.73:0/1834045918 conn(0x7f57040fc880 msgr2=0x7f57041057a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 shutdown_connections 2024-03-09T00:06:31.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.336+0000 7f570a4bf640 1 -- 172.21.15.73:0/1834045918 wait complete. 2024-03-09T00:06:31.341 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1866373286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:31.341 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2632701611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.341 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1053901261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.341 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/652061149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.342 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:31.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 -- 172.21.15.73:0/1188320337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c075330 msgr2=0x7f3e0c073780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 --2- 172.21.15.73:0/1188320337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c075330 0x7f3e0c073780 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009900 tx=0x7f3e0802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:31.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 -- 172.21.15.73:0/1188320337 shutdown_connections 2024-03-09T00:06:31.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 --2- 172.21.15.73:0/1188320337 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c1091c0 0x7f3e0c10b650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 --2- 172.21.15.73:0/1188320337 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c073d50 0x7f3e0c0741d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 --2- 172.21.15.73:0/1188320337 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c075330 0x7f3e0c073780 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 -- 172.21.15.73:0/1188320337 >> 172.21.15.73:0/1188320337 conn(0x7f3e0c0fc8d0 msgr2=0x7f3e0c0fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 -- 172.21.15.73:0/1188320337 shutdown_connections 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.449+0000 7f3e13360640 1 -- 172.21.15.73:0/1188320337 wait complete. 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 Processor -- start 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 -- start start 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 0x7f3e0c199d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1091c0 0x7f3e0c1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c10f180 con 0x7f3e0c073d50 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c10f000 con 0x7f3e0c1091c0 2024-03-09T00:06:31.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e13360640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0c10f300 con 0x7f3e0c075330 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49286/0 (socket says 172.21.15.73:49286) 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 -- 172.21.15.73:0/1189152415 learned_addr learned my addr 172.21.15.73:0/1189152415 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e118fd640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1091c0 0x7f3e0c1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e108fb640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 0x7f3e0c199d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 -- 172.21.15.73:0/1189152415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 msgr2=0x7f3e0c199d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 0x7f3e0c199d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 -- 172.21.15.73:0/1189152415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1091c0 msgr2=0x7f3e0c1a0da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1091c0 0x7f3e0c1a0da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 -- 172.21.15.73:0/1189152415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0c1a1410 con 0x7f3e0c073d50 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e108fb640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 0x7f3e0c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e110fc640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009230 tx=0x7f3e08005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.450+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08041070 con 0x7f3e0c073d50 2024-03-09T00:06:31.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.451+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e080054e0 con 0x7f3e0c073d50 2024-03-09T00:06:31.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.451+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0c1a16a0 con 0x7f3e0c073d50 2024-03-09T00:06:31.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.451+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e08034440 con 0x7f3e0c073d50 2024-03-09T00:06:31.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.451+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0c1a1b90 con 0x7f3e0c073d50 2024-03-09T00:06:31.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.451+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e0c107ef0 con 0x7f3e0c073d50 2024-03-09T00:06:31.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.453+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3e08044070 con 0x7f3e0c073d50 2024-03-09T00:06:31.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.453+0000 7f3e027fc640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 0x7f3df80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.453+0000 7f3e108fb640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 0x7f3df80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.453+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3e080f0ff0 con 0x7f3e0c073d50 2024-03-09T00:06:31.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.454+0000 7f3e108fb640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 0x7f3df80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c19ad00 tx=0x7f3dfc005d30 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.457+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3e080bdce0 con 0x7f3e0c073d50 2024-03-09T00:06:31.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1866373286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:31.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2632701611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1053901261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/652061149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1866373286' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2632701611' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1053901261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/652061149' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:31.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.667+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e0c02ce70 con 0x7f3e0c073d50 2024-03-09T00:06:31.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.668+0000 7f3e027fc640 1 -- 172.21.15.73:0/1189152415 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f3e080bd5e0 con 0x7f3e0c073d50 2024-03-09T00:06:31.663 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:31.663 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:31.664 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:31.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 msgr2=0x7f3df80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 0x7f3df80a4cc0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f3e0c19ad00 tx=0x7f3dfc005d30 comp rx=0 tx=0).stop 2024-03-09T00:06:31.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 msgr2=0x7f3e0c1997f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x7f3e08009230 tx=0x7f3e08005dd0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 shutdown_connections 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df80a2800 0x7f3df80a4cc0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e0c1091c0 0x7f3e0c1a0da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e0c075330 0x7f3e0c199d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 --2- 172.21.15.73:0/1189152415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e0c073d50 0x7f3e0c1997f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 >> 172.21.15.73:0/1189152415 conn(0x7f3e0c0fc8d0 msgr2=0x7f3e0c1050c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 shutdown_connections 2024-03-09T00:06:31.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.670+0000 7f3e13360640 1 -- 172.21.15.73:0/1189152415 wait complete. 2024-03-09T00:06:31.677 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:31.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.783+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2982310400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b0101da0 msgr2=0x7f24b0104190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.783+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2982310400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b0101da0 0x7f24b0104190 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f24a8009ca0 tx=0x7f24a802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.783+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2982310400 shutdown_connections 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2982310400 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0107000 0x7f24b0109450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2982310400 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b01046d0 0x7f24b0106ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2982310400 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b0101da0 0x7f24b0104190 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2982310400 >> 172.21.15.73:0/2982310400 conn(0x7f24b00fb7d0 msgr2=0x7f24b00fdbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2982310400 shutdown_connections 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2982310400 wait complete. 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 Processor -- start 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- start start 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0101da0 0x7f24b0197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b0107000 0x7f24b019ebb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b010cfb0 con 0x7f24b01046d0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b010ce30 con 0x7f24b0107000 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b6ba9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b010d130 con 0x7f24b0101da0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49310/0 (socket says 172.21.15.73:49310) 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24affff640 1 -- 172.21.15.73:0/2933333560 learned_addr learned my addr 172.21.15.73:0/2933333560 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.784+0000 7f24b4945640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0101da0 0x7f24b0197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 -- 172.21.15.73:0/2933333560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0101da0 msgr2=0x7f24b0197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0101da0 0x7f24b0197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24b5146640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b0107000 0x7f24b019ebb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 -- 172.21.15.73:0/2933333560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b0107000 msgr2=0x7f24b019ebb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b0107000 0x7f24b019ebb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 -- 172.21.15.73:0/2933333560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b019f220 con 0x7f24b01046d0 2024-03-09T00:06:31.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24affff640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9c0 tx=0x7f24a400ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a400cd80 con 0x7f24b01046d0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24a4005500 con 0x7f24b01046d0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b019f510 con 0x7f24b01046d0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24a4010690 con 0x7f24b01046d0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.785+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b019fab0 con 0x7f24b01046d0 2024-03-09T00:06:31.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.786+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b00fd2c0 con 0x7f24b01046d0 2024-03-09T00:06:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.787+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f24a40046c0 con 0x7f24b01046d0 2024-03-09T00:06:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.788+0000 7f24adffb640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 0x7f24980a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.788+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f24a40cf1a0 con 0x7f24b01046d0 2024-03-09T00:06:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.788+0000 7f24b4945640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 0x7f24980a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:31.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.788+0000 7f24b4945640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 0x7f24980a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24a802c700 tx=0x7f24a80323c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:31.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.793+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f24a409be90 con 0x7f24b01046d0 2024-03-09T00:06:31.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.966+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f24b0061ef0 con 0x7f24b01046d0 2024-03-09T00:06:31.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.967+0000 7f24adffb640 1 -- 172.21.15.73:0/2933333560 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7f24a409b790 con 0x7f24b01046d0 2024-03-09T00:06:31.962 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:31.963 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:31.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 msgr2=0x7f24980a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 0x7f24980a4d40 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f24a802c700 tx=0x7f24a80323c0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 msgr2=0x7f24b0197b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f24a400e9c0 tx=0x7f24a400ee80 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 shutdown_connections 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f24980a2880 0x7f24980a4d40 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f24b0107000 0x7f24b019ebb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f24b01046d0 0x7f24b0197b40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 --2- 172.21.15.73:0/2933333560 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f24b0101da0 0x7f24b0197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 >> 172.21.15.73:0/2933333560 conn(0x7f24b00fb7d0 msgr2=0x7f24b0102640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 shutdown_connections 2024-03-09T00:06:31.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:31.970+0000 7f24b6ba9640 1 -- 172.21.15.73:0/2933333560 wait complete. 2024-03-09T00:06:31.976 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-09T00:04:13.661086+0000', 'modified': '2024-03-09T00:04:39.321347+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 24515}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_24493': {'gid': 24493, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.179:6839/2754171892', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.179:6838', 'nonce': 2754171892}, {'type': 'v1', 'addr': '172.21.15.179:6839', 'nonce': 2754171892}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.102:6839/3264474111', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.102:6838', 'nonce': 3264474111}, {'type': 'v1', 'addr': '172.21.15.102:6839', 'nonce': 3264474111}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3, 4], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0} 2024-03-09T00:06:31.977 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2024-03-09T00:06:31.977 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 -- 172.21.15.73:0/1169713656 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 msgr2=0x7fce60104170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 --2- 172.21.15.73:0/1169713656 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60104170 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7fce50009900 tx=0x7fce5002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 -- 172.21.15.73:0/1169713656 shutdown_connections 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 --2- 172.21.15.73:0/1169713656 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce60106fe0 0x7fce60109430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 --2- 172.21.15.73:0/1169713656 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce601046b0 0x7fce60106aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 --2- 172.21.15.73:0/1169713656 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60104170 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 -- 172.21.15.73:0/1169713656 >> 172.21.15.73:0/1169713656 conn(0x7fce600fb790 msgr2=0x7fce600fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 -- 172.21.15.73:0/1169713656 shutdown_connections 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.086+0000 7fce66843640 1 -- 172.21.15.73:0/1169713656 wait complete. 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 Processor -- start 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 -- start start 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 0x7fce60199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce60106fe0 0x7fce601a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60100d10 con 0x7fce60101d80 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60100b90 con 0x7fce60106fe0 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce66843640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce60100e90 con 0x7fce601046b0 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49318/0 (socket says 172.21.15.73:49318) 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 -- 172.21.15.73:0/1195970763 learned_addr learned my addr 172.21.15.73:0/1195970763 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 0x7fce60199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 -- 172.21.15.73:0/1195970763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 msgr2=0x7fce60199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 0x7fce60199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 -- 172.21.15.73:0/1195970763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce60106fe0 msgr2=0x7fce601a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce64de0640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce60106fe0 0x7fce601a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce60106fe0 0x7fce601a0e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5ffff640 1 -- 172.21.15.73:0/1195970763 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce601a14b0 con 0x7fce60101d80 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.087+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 0x7fce60199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce50009230 tx=0x7fce50005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce5d7fa640 1 -- 172.21.15.73:0/1195970763 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50041070 con 0x7fce60101d80 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce5d7fa640 1 -- 172.21.15.73:0/1195970763 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce500054e0 con 0x7fce60101d80 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce5d7fa640 1 -- 172.21.15.73:0/1195970763 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce50034440 con 0x7fce60101d80 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce601a1740 con 0x7fce60101d80 2024-03-09T00:06:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.088+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce601a1c30 con 0x7fce60101d80 2024-03-09T00:06:32.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.090+0000 7fce5d7fa640 1 -- 172.21.15.73:0/1195970763 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fce50044070 con 0x7fce60101d80 2024-03-09T00:06:32.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.090+0000 7fce6941a740 1 Processor -- start 2024-03-09T00:06:32.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.090+0000 7fce6941a740 1 -- start start 2024-03-09T00:06:32.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5d7fa640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 0x7fce480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6a4500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a9c40 0x5567bf6ac0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6097c0 con 0x5567bf6a4ad0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf5b2b50 con 0x5567bf6a9c40 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce6941a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf605860 con 0x5567bf6a40e0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce64de0640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 0x7fce480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5d7fa640 1 -- 172.21.15.73:0/1195970763 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fce500f0bd0 con 0x7fce60101d80 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49326/0 (socket says 172.21.15.73:49326) 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 -- 172.21.15.73:0/1063917965 learned_addr learned my addr 172.21.15.73:0/1063917965 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6a4500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce64de0640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a9c40 0x5567bf6ac0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 -- 172.21.15.73:0/1063917965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 msgr2=0x5567bf6a4500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6a4500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 -- 172.21.15.73:0/1063917965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a9c40 msgr2=0x5567bf6ac0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a9c40 0x5567bf6ac0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 -- 172.21.15.73:0/1063917965 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567bf5c69a0 con 0x5567bf6a4ad0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce64de0640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 0x7fce480a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce580092f0 tx=0x7fce580137d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce5400bd00 tx=0x7fce5400f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42994cb4b291d496 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce467fc640 1 -- 172.21.15.73:0/1063917965 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce54005020 con 0x5567bf6a4ad0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce467fc640 1 -- 172.21.15.73:0/1063917965 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce540047a0 con 0x5567bf6a4ad0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.091+0000 7fce467fc640 1 -- 172.21.15.73:0/1063917965 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce540116a0 con 0x5567bf6a4ad0 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- 172.21.15.73:0/1063917965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 msgr2=0x5567bf6a95d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fce5400bd00 tx=0x7fce5400f730 comp rx=0 tx=0).stop 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- 172.21.15.73:0/1063917965 shutdown_connections 2024-03-09T00:06:32.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a9c40 0x5567bf6ac0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a4ad0 0x5567bf6a95d0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 --2- 172.21.15.73:0/1063917965 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6a4500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- 172.21.15.73:0/1063917965 >> 172.21.15.73:0/1063917965 conn(0x5567bf66c400 msgr2=0x5567bf69fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- 172.21.15.73:0/1063917965 shutdown_connections 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- 172.21.15.73:0/1063917965 wait complete. 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.092+0000 7fce6941a740 1 -- start start 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 Processor -- start 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6ae620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6aeb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6acbf0 con 0x5567bf6a9c40 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf5c5f90 con 0x5567bf6a4ad0 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6acd70 con 0x5567bf6a40e0 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce6941a740 10 client.0 _ll_register_callbacks cb 0x5567bf601400 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:32.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49342/0 (socket says 172.21.15.73:49342) 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 -- 172.21.15.73:0/2527278725 learned_addr learned my addr 172.21.15.73:0/2527278725 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6ae620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 -- 172.21.15.73:0/2527278725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 msgr2=0x5567bf6ae620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6ae620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 -- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 msgr2=0x5567bf6aeb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6aeb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6aeb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 -- 172.21.15.73:0/2527278725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567bf31e2d0 con 0x5567bf6a9c40 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce64de0640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fce58011af0 tx=0x7fce58004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46917fdcda7e1a38 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.093+0000 7fce3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce58023030 con 0x5567bf6a9c40 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fce5800fbc0 con 0x5567bf6a9c40 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce540047a0 con 0x5567bf6a9c40 2024-03-09T00:06:32.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce58030050 con 0x5567bf6a9c40 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 10 client.15366 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5567bf3e86b0 con 0x5567bf6a9c40 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce540047a0 con 0x5567bf6a9c40 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fce5800f050 con 0x5567bf6a9c40 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 10 client.15366 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 10 client.15366 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 20 client.15366 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6bb250 0x5567bf6ba270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 4 client.15366 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5567bf616d00 con 0x5567bf6bb250 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.094+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6bb250 0x5567bf6ba270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.095+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6bb250 0x5567bf6ba270 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1ab3dbfa418f22f4 server_cookie=c3227f330f133664 in_seq=0 out_seq=0 2024-03-09T00:06:32.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.095+0000 7fce3f7fe640 10 client.15366 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.095+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2527278725 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5567bf616d00 con 0x5567bf6bb250 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.095+0000 7fce3f7fe640 10 client.15366 handle_command_reply: tid=0 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.095+0000 7fce6941a740 1 client.15366 shutdown 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 msgr2=0x5567bf6b5ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fce58011af0 tx=0x7fce58004ab0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 shutdown_connections 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6bb250 0x5567bf6ba270 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf6a9c40 0x5567bf6b5ba0 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6aeb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce3f7fe640 0 client.15366 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 --2- 172.21.15.73:0/2527278725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6ae620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 >> 172.21.15.73:0/2527278725 conn(0x5567bf66c400 msgr2=0x5567bf69ec90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 shutdown_connections 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 1 -- 172.21.15.73:0/2527278725 wait complete. 2024-03-09T00:06:32.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.096+0000 7fce6941a740 20 client.15366 trim_cache size 0 max 16384 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 Processor -- start 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- start start 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b5b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf69e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6baa70 con 0x5567bf66c400 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6bb0d0 con 0x5567bf6a4ad0 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6bb730 con 0x5567bf6a40e0 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49352/0 (socket says 172.21.15.73:49352) 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 -- 172.21.15.73:0/1603409740 learned_addr learned my addr 172.21.15.73:0/1603409740 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:32.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b5b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 -- 172.21.15.73:0/1603409740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 msgr2=0x5567bf6b5b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b5b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 -- 172.21.15.73:0/1603409740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 msgr2=0x5567bf69e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf69e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 -- 172.21.15.73:0/1603409740 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5567bf69fff0 con 0x5567bf66c400 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce5ffff640 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce5002cfd0 tx=0x7fce50009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96b98693018085a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce45ffb640 1 -- 172.21.15.73:0/1603409740 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce5004b040 con 0x5567bf66c400 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce45ffb640 1 -- 172.21.15.73:0/1603409740 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce50004c60 con 0x5567bf66c400 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce45ffb640 1 -- 172.21.15.73:0/1603409740 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500c2a10 con 0x5567bf66c400 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- 172.21.15.73:0/1603409740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 msgr2=0x5567bf69e400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fce5002cfd0 tx=0x7fce50009260 comp rx=0 tx=0).stop 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 -- 172.21.15.73:0/1603409740 shutdown_connections 2024-03-09T00:06:32.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf69e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b5b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.112+0000 7fce6941a740 1 --2- 172.21.15.73:0/1603409740 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf69e400 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- 172.21.15.73:0/1603409740 >> 172.21.15.73:0/1603409740 conn(0x5567bf69f940 msgr2=0x5567bf6ae620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- 172.21.15.73:0/1603409740 shutdown_connections 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- 172.21.15.73:0/1603409740 wait complete. 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- start start 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 Processor -- start 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b37f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6b4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6bc3f0 con 0x5567bf66c400 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6bbd90 con 0x5567bf6a4ad0 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5567bf6bca50 con 0x5567bf6a40e0 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce6941a740 10 client.0 _ll_register_callbacks cb 0x7fce5400b4c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:32.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce5ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49366/0 (socket says 172.21.15.73:49366) 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce5ffff640 1 -- 172.21.15.73:0/2011201101 learned_addr learned my addr 172.21.15.73:0/2011201101 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b37f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.113+0000 7fce64de0640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6b4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 -- 172.21.15.73:0/2011201101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 msgr2=0x5567bf6b37f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b37f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 -- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 msgr2=0x5567bf6b4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6b4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 -- 172.21.15.73:0/2011201101 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5567bf6b4d80 con 0x5567bf66c400 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce64de0640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce5ffff640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce50006010 tx=0x7fce50009680 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bc1adeec1fe029f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce500d5070 con 0x5567bf66c400 2024-03-09T00:06:32.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fce500bb4c0 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce50004c60 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce5004b070 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 10 client.15372 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5567bf712e60 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce50004c60 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fce500c54d0 con 0x5567bf66c400 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 10 client.15372 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 10 client.15372 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 20 client.15372 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6a9c40 0x5567bf713030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 4 client.15372 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.114+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5567bf611050 con 0x5567bf6a9c40 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6a9c40 0x5567bf713030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:32.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce5f7fe640 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6a9c40 0x5567bf713030 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=74193914f781fe67 server_cookie=a223c0f7aa17218c in_seq=0 out_seq=0 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce3f7fe640 10 client.15372 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce3f7fe640 1 -- 172.21.15.73:0/2011201101 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x5567bf611050 con 0x5567bf6a9c40 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce3f7fe640 10 client.15372 handle_command_reply: tid=0 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 client.15372 shutdown 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 msgr2=0x5567bf6b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fce50006010 tx=0x7fce50009680 comp rx=0 tx=0).stop 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 shutdown_connections 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5567bf6a9c40 0x5567bf713030 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5567bf6a4ad0 0x5567bf6b4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce3f7fe640 0 client.15372 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5567bf6a40e0 0x5567bf6b37f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 --2- 172.21.15.73:0/2011201101 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5567bf66c400 0x5567bf6b32b0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 >> 172.21.15.73:0/2011201101 conn(0x5567bf69f940 msgr2=0x5567bf6a2c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 shutdown_connections 2024-03-09T00:06:32.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 1 -- 172.21.15.73:0/2011201101 wait complete. 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.115+0000 7fce6941a740 20 client.15372 trim_cache size 0 max 16384 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "cluster_fsid": "44f886a8-dda7-11ee-95c6-87774f69a715", 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "whoami": 0, 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "id": 24515, 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "want_state": "up:active", 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "state": "up:active", 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "fs_name": "cephfs", 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "rank_uptime": 137.79396437099999, 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "mdsmap_epoch": 20, 2024-03-09T00:06:32.118 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch": 93, 2024-03-09T00:06:32.119 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_epoch_barrier": 89, 2024-03-09T00:06:32.119 INFO:teuthology.orchestra.run.smithi073.stdout: "uptime": 143.200461152 2024-03-09T00:06:32.119 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:06:32.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 msgr2=0x7fce480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 0x7fce480a4cc0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fce580092f0 tx=0x7fce580137d0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 msgr2=0x7fce60199890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fce50009230 tx=0x7fce50005dd0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 shutdown_connections 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fce480a2800 0x7fce480a4cc0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce60106fe0 0x7fce601a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce601046b0 0x7fce60199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 --2- 172.21.15.73:0/1195970763 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce60101d80 0x7fce60199890 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 >> 172.21.15.73:0/1195970763 conn(0x7fce600fb790 msgr2=0x7fce60107420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 shutdown_connections 2024-03-09T00:06:32.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:32.116+0000 7fce66843640 1 -- 172.21.15.73:0/1195970763 wait complete. 2024-03-09T00:06:32.122 INFO:teuthology.run_tasks:Running task kclient... 2024-03-09T00:06:32.235 INFO:tasks.kclient:Mounting kernel clients... 2024-03-09T00:06:32.235 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-09T00:06:32.235 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:32.235 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-03-09T00:06:32.251 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-ns--home-ubuntu-cephtest-mnt.admin (id: 0) 2024-03-09T00:06:32.251 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:32.251 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.admin 2024-03-09T00:06:32.314 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:32 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1834045918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:32.314 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:32 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1189152415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.314 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:32 smithi073 ceph-mon[32309]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:32.314 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:32 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2933333560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.317 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:32.318 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link delete ceph-brx 2024-03-09T00:06:32.412 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=legacy', 'nowsync'], 'syntax': 'v1', 'mount_subvol_num': 1} 2024-03-09T00:06:32.412 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-09T00:06:32.413 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:32.413 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph fs ls 2024-03-09T00:06:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:32 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1834045918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:32 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1189152415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:32 smithi102 ceph-mon[36106]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:32 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2933333560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:32 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1834045918' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:32.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:32 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1189152415' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:32 smithi179 ceph-mon[36637]: pgmap v367: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:32.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:32 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2933333560' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:32.656 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:06:34.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.460+0000 7f8276fe9700 1 -- 172.21.15.73:0/1997163208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074930 msgr2=0x7f827010e2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.460+0000 7f8276fe9700 1 --2- 172.21.15.73:0/1997163208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074930 0x7f827010e2a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 crypto rx=0x7f825c009990 tx=0x7f825c009ca0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 -- 172.21.15.73:0/1997163208 shutdown_connections 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 --2- 172.21.15.73:0/1997163208 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 0x7f8270110bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 --2- 172.21.15.73:0/1997163208 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074010 0x7f82700743f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 --2- 172.21.15.73:0/1997163208 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074930 0x7f827010e2a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 -- 172.21.15.73:0/1997163208 >> 172.21.15.73:0/1997163208 conn(0x7f8270078f10 msgr2=0x7f8270079320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 -- 172.21.15.73:0/1997163208 shutdown_connections 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 -- 172.21.15.73:0/1997163208 wait complete. 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 Processor -- start 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.461+0000 7f8276fe9700 1 -- start start 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:34.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074930 0x7f82701a0920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 0x7f827019a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270114310 con 0x7f8270074010 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270114190 con 0x7f827010e7e0 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8270114490 con 0x7f8270074930 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49384/0 (socket says 172.21.15.73:49384) 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 -- 172.21.15.73:0/2126202513 learned_addr learned my addr 172.21.15.73:0/2126202513 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8275586700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 0x7f827019a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f826ffff700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074930 0x7f82701a0920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 -- 172.21.15.73:0/2126202513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074930 msgr2=0x7f82701a0920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074930 0x7f82701a0920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 -- 172.21.15.73:0/2126202513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 msgr2=0x7f827019a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 0x7f827019a5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 -- 172.21.15.73:0/2126202513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f827019ad50 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8274d85700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f826400c970 tx=0x7f826400cc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8264007aa0 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8264004d10 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827019b040 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.462+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f826400f450 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.463+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82701a72b0 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.464+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8270068aa0 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.465+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f8264004750 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.465+0000 7f826dffb700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 0x7f82600a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.465+0000 7f826ffff700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 0x7f82600a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.465+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f82640ceba0 con 0x7f8270074010 2024-03-09T00:06:34.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.466+0000 7f826ffff700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 0x7f82600a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f825c006010 tx=0x7f825c00b540 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:34.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.469+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f826409b920 con 0x7f8270074010 2024-03-09T00:06:34.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.709+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7f827019bad0 con 0x7f8270074010 2024-03-09T00:06:34.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.709+0000 7f826dffb700 1 -- 172.21.15.73:0/2126202513 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v20) v1 ==== 53+0+88 (secure 0 0 0) 0x7f8264005700 con 0x7f8270074010 2024-03-09T00:06:34.704 INFO:teuthology.orchestra.run.smithi073.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data cephfs_data_ec ] 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 msgr2=0x7f82600a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 0x7f82600a4d50 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f825c006010 tx=0x7f825c00b540 comp rx=0 tx=0).stop 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 msgr2=0x7f82701a03e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 crypto rx=0x7f826400c970 tx=0x7f826400cc80 comp rx=0 tx=0).stop 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 shutdown_connections 2024-03-09T00:06:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8270074010 0x7f82701a03e0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f82600a2890 0x7f82600a4d50 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f827010e7e0 0x7f827019a5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 --2- 172.21.15.73:0/2126202513 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8270074930 0x7f82701a0920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.711+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 >> 172.21.15.73:0/2126202513 conn(0x7f8270078f10 msgr2=0x7f82700feff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.712+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 shutdown_connections 2024-03-09T00:06:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:34.712+0000 7f8276fe9700 1 -- 172.21.15.73:0/2126202513 wait complete. 2024-03-09T00:06:35.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:35 smithi179 ceph-mon[36637]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:35.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2126202513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:06:35.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:35 smithi073 ceph-mon[32309]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:35.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2126202513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:06:35.404 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-09T00:06:35.404 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi073.front.sepia.ceph.com 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-09T00:06:35.405 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-09T00:06:35.405 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.405 DEBUG:teuthology.orchestra.run.smithi073:> ip addr 2024-03-09T00:06:35.421 INFO:teuthology.orchestra.run.smithi073.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-09T00:06:35.421 INFO:teuthology.orchestra.run.smithi073.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-09T00:06:35.421 INFO:teuthology.orchestra.run.smithi073.stdout: inet 127.0.0.1/8 scope host lo 2024-03-09T00:06:35.421 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:06:35.421 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 ::1/128 scope host 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:ea brd ff:ff:ff:ff:ff:ff 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f0 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:d8:d9:eb brd ff:ff:ff:ff:ff:ff 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: altname enp7s0f1 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7c brd ff:ff:ff:ff:ff:ff 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: link/ether 0c:c4:7a:8f:ce:7d brd ff:ff:ff:ff:ff:ff 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: inet 172.21.15.73/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-09T00:06:35.422 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft 42082sec preferred_lft 42082sec 2024-03-09T00:06:35.423 INFO:teuthology.orchestra.run.smithi073.stdout: inet6 fe80::ec4:7aff:fe8f:ce7d/64 scope link 2024-03-09T00:06:35.423 INFO:teuthology.orchestra.run.smithi073.stdout: valid_lft forever preferred_lft forever 2024-03-09T00:06:35.423 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-09T00:06:35.423 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.423 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:06:35.423 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:06:35.423 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add name ceph-brx type bridge 2024-03-09T00:06:35.424 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr flush dev ceph-brx 2024-03-09T00:06:35.424 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set ceph-brx up 2024-03-09T00:06:35.424 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-09T00:06:35.424 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:06:35.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:35 smithi102 ceph-mon[36106]: pgmap v368: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:35.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2126202513' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-09T00:06:35.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.532 DEBUG:teuthology.orchestra.run.smithi073:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-09T00:06:35.558 INFO:teuthology.orchestra.run.smithi073.stdout:1 2024-03-09T00:06:35.560 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.560 DEBUG:teuthology.orchestra.run.smithi073:> route 2024-03-09T00:06:35.615 INFO:teuthology.orchestra.run.smithi073.stdout:Kernel IP routing table 2024-03-09T00:06:35.616 INFO:teuthology.orchestra.run.smithi073.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-09T00:06:35.616 INFO:teuthology.orchestra.run.smithi073.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-09T00:06:35.616 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-09T00:06:35.616 INFO:teuthology.orchestra.run.smithi073.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-09T00:06:35.616 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.616 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:06:35.616 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:06:35.616 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-09T00:06:35.616 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-09T00:06:35.617 DEBUG:teuthology.orchestra.run.smithi073:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-09T00:06:35.617 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:06:35.711 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.712 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-03-09T00:06:35.727 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.727 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list-id 2024-03-09T00:06:35.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.782 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:06:35.782 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:06:35.782 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-09T00:06:35.782 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-09T00:06:35.782 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:06:35.864 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-09T00:06:35.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-09T00:06:35.864 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:06:35.989 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:35.990 DEBUG:teuthology.orchestra.run.smithi073:> (cd / && exec bash -c ' 2024-03-09T00:06:35.990 DEBUG:teuthology.orchestra.run.smithi073:> set -e 2024-03-09T00:06:35.990 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set brx.0 up 2024-03-09T00:06:35.990 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link set dev brx.0 master ceph-brx 2024-03-09T00:06:35.990 DEBUG:teuthology.orchestra.run.smithi073:> ') 2024-03-09T00:06:36.032 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-09T00:06:36.033 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:36.087 INFO:teuthology.orchestra.run.smithi073.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-09T00:06:36.087 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-09T00:06:36.088 DEBUG:teuthology.orchestra.run.smithi073:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:36.142 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-09T00:06:36.142 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-09T00:06:36.142 INFO:tasks.cephfs.kernel_mount:mounting using device: :/volumes/qa/sv_1/e1c20de2-a1eb-4247-9eda-1427c89645a8 2024-03-09T00:06:36.142 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.142 DEBUG:teuthology.orchestra.run.smithi073:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/volumes/qa/sv_1/e1c20de2-a1eb-4247-9eda-1427c89645a8 /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-09T00:06:36.226 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-09T00:06:36.226 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-09T00:06:36.226 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync". 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:invalid new device string format 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: resolved to: "172.21.15.73:6789,172.21.15.102:6789,172.21.15.179:6789" 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.73:6789,172.21.15.102:6789,172.21.15.179:6789:/volumes/qa/sv_1/e1c20de2-a1eb-4247-9eda-1427c89645a8 2024-03-09T00:06:36.346 INFO:teuthology.orchestra.run.smithi073.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,ms_mode=legacy,nowsync,key=0,fsid=44f886a8-dda7-11ee-95c6-87774f69a715" will pass to kernel 2024-03-09T00:06:36.347 INFO:teuthology.orchestra.run.smithi073.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-09T00:06:36.347 INFO:teuthology.orchestra.run.smithi073.stdout: You just mounted a file system that supports labels which does not 2024-03-09T00:06:36.347 INFO:teuthology.orchestra.run.smithi073.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-09T00:06:36.347 INFO:teuthology.orchestra.run.smithi073.stdout: applications will generate AVC messages and not be allowed access to 2024-03-09T00:06:36.347 INFO:teuthology.orchestra.run.smithi073.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-09T00:06:36.349 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-09T00:06:36.349 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.349 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:36.417 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.418 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-09T00:06:36.474 INFO:teuthology.orchestra.run.smithi073.stdout:client25036 2024-03-09T00:06:36.474 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:06:36.474 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-09T00:06:36.532 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.532 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-09T00:06:36.588 INFO:teuthology.orchestra.run.smithi073.stdout:client25036 2024-03-09T00:06:36.588 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.589 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-09T00:06:36.645 INFO:teuthology.orchestra.run.smithi073.stdout:client25036 2024-03-09T00:06:36.645 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:36.645 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/44f886a8-dda7-11ee-95c6-87774f69a715.client25036/status') 2024-03-09T00:06:36.711 INFO:teuthology.orchestra.run.smithi073.stdout:instance: client.25036 (3)172.21.15.73:0/4237484083 2024-03-09T00:06:36.712 INFO:teuthology.orchestra.run.smithi073.stdout:blocklisted: false 2024-03-09T00:06:36.712 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records in 2024-03-09T00:06:36.712 INFO:teuthology.orchestra.run.smithi073.stderr:0+1 records out 2024-03-09T00:06:36.712 INFO:teuthology.orchestra.run.smithi073.stderr:71 bytes copied, 2.0779e-05 s, 3.4 MB/s 2024-03-09T00:06:36.713 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-09T00:06:36.746 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-09T00:06:36.775 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:36.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 -- 172.21.15.73:0/428998686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050101de0 msgr2=0x7f5050108670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 --2- 172.21.15.73:0/428998686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050101de0 0x7f5050108670 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7f5048009ca0 tx=0x7f504802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 -- 172.21.15.73:0/428998686 shutdown_connections 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 --2- 172.21.15.73:0/428998686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050101de0 0x7f5050108670 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 --2- 172.21.15.73:0/428998686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 0x7f50501018a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 --2- 172.21.15.73:0/428998686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050100230 0x7f5050100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 -- 172.21.15.73:0/428998686 >> 172.21.15.73:0/428998686 conn(0x7f50500fb7a0 msgr2=0x7f50500fdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 -- 172.21.15.73:0/428998686 shutdown_connections 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.883+0000 7f50568cd640 1 -- 172.21.15.73:0/428998686 wait complete. 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 Processor -- start 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 -- start start 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:36.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 0x7f505019c380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 0x7f505019e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505010bf40 con 0x7f5050100230 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505010bdc0 con 0x7f5050101de0 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f50568cd640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505010c0c0 con 0x7f5050101420 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49394/0 (socket says 172.21.15.73:49394) 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 -- 172.21.15.73:0/1670315745 learned_addr learned my addr 172.21.15.73:0/1670315745 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504f7fe640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 0x7f505019c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 -- 172.21.15.73:0/1670315745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 msgr2=0x7f505019c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:36.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 0x7f505019c380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f5054e6a640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 0x7f505019e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.884+0000 7f504ffff640 1 -- 172.21.15.73:0/1670315745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 msgr2=0x7f505019e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504ffff640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 0x7f505019e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504ffff640 1 -- 172.21.15.73:0/1670315745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f505019f280 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f5054e6a640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 0x7f505019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504ffff640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f504000b450 tx=0x7f504000b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5040015020 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f505019f570 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5040011d70 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f504000f340 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.885+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f505019fac0 con 0x7f5050100230 2024-03-09T00:06:36.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.886+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5050107550 con 0x7f5050100230 2024-03-09T00:06:36.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.887+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5040018020 con 0x7f5050100230 2024-03-09T00:06:36.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.887+0000 7f504d7fa640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 0x7f50380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:36.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.887+0000 7f504f7fe640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 0x7f50380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:36.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.887+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f50400ce970 con 0x7f5050100230 2024-03-09T00:06:36.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.888+0000 7f504f7fe640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 0x7f50380a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5044005fd0 tx=0x7f5044005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:36.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:36.892+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f504001f020 con 0x7f5050100230 2024-03-09T00:06:37.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.070+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5050061ef0 con 0x7f5050100230 2024-03-09T00:06:37.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.071+0000 7f504d7fa640 1 -- 172.21.15.73:0/1670315745 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7f504000d040 con 0x7f5050100230 2024-03-09T00:06:37.066 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:37.066 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.073+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 msgr2=0x7f50380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.073+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 0x7f50380a4cc0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f5044005fd0 tx=0x7f5044005850 comp rx=0 tx=0).stop 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.073+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 msgr2=0x7f505006a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.073+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7f504000b450 tx=0x7f504000b910 comp rx=0 tx=0).stop 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 shutdown_connections 2024-03-09T00:06:37.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f50380a2800 0x7f50380a4cc0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5050101de0 0x7f505019e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5050101420 0x7f505019c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 --2- 172.21.15.73:0/1670315745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5050100230 0x7f505006a4b0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 >> 172.21.15.73:0/1670315745 conn(0x7f50500fb7a0 msgr2=0x7f5050104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 shutdown_connections 2024-03-09T00:06:37.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.074+0000 7f50568cd640 1 -- 172.21.15.73:0/1670315745 wait complete. 2024-03-09T00:06:37.080 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:37.081 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:37.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 -- 172.21.15.73:0/2021981722 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c102ec0 msgr2=0x7f0e2c109750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 --2- 172.21.15.73:0/2021981722 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c109750 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7f0e24009ca0 tx=0x7f0e2402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 -- 172.21.15.73:0/2021981722 shutdown_connections 2024-03-09T00:06:37.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 --2- 172.21.15.73:0/2021981722 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c109750 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 --2- 172.21.15.73:0/2021981722 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102500 0x7f0e2c102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 --2- 172.21.15.73:0/2021981722 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c101310 0x7f0e2c101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 -- 172.21.15.73:0/2021981722 >> 172.21.15.73:0/2021981722 conn(0x7f0e2c0fc860 msgr2=0x7f0e2c0fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 -- 172.21.15.73:0/2021981722 shutdown_connections 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.191+0000 7f0e334c0640 1 -- 172.21.15.73:0/2021981722 wait complete. 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 Processor -- start 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 -- start start 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c102500 0x7f0e2c197bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10d080 con 0x7f0e2c101310 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10cf00 con 0x7f0e2c102500 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e334c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e2c10d200 con 0x7f0e2c102ec0 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49412/0 (socket says 172.21.15.73:49412) 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 -- 172.21.15.73:0/408447680 learned_addr learned my addr 172.21.15.73:0/408447680 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e30a5b640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c102500 0x7f0e2c197bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e31a5d640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 -- 172.21.15.73:0/408447680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 msgr2=0x7f0e2c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 -- 172.21.15.73:0/408447680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c102500 msgr2=0x7f0e2c197bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c102500 0x7f0e2c197bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 -- 172.21.15.73:0/408447680 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e2c19f2b0 con 0x7f0e2c101310 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e31a5d640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.192+0000 7f0e3125c640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b450 tx=0x7f0e2800b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.193+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e28015020 con 0x7f0e2c101310 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.193+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0e28011d70 con 0x7f0e2c101310 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.193+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e2c19f5a0 con 0x7f0e2c101310 2024-03-09T00:06:37.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.193+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0e2800f340 con 0x7f0e2c101310 2024-03-09T00:06:37.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.193+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e2c19fbc0 con 0x7f0e2c101310 2024-03-09T00:06:37.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.194+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e2c108630 con 0x7f0e2c101310 2024-03-09T00:06:37.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.195+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f0e28018020 con 0x7f0e2c101310 2024-03-09T00:06:37.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.195+0000 7f0e227fc640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 0x7f0e180a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.195+0000 7f0e30a5b640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 0x7f0e180a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.196+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f0e280cedf0 con 0x7f0e2c101310 2024-03-09T00:06:37.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.196+0000 7f0e30a5b640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 0x7f0e180a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c198ba0 tx=0x7f0e1c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.199+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f0e2801f020 con 0x7f0e2c101310 2024-03-09T00:06:37.292 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:37 smithi073 ceph-mon[32309]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:37.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:37 smithi179 ceph-mon[36637]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:37.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.381+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f0e2c061ef0 con 0x7f0e2c101310 2024-03-09T00:06:37.376 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.382+0000 7f0e227fc640 1 -- 172.21.15.73:0/408447680 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f0e2800d040 con 0x7f0e2c101310 2024-03-09T00:06:37.377 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 msgr2=0x7f0e180a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 0x7f0e180a4d40 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f0e2c198ba0 tx=0x7f0e1c008040 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 msgr2=0x7f0e2c197690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7f0e2800b450 tx=0x7f0e2800b910 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 shutdown_connections 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f0e180a2880 0x7f0e180a4d40 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e2c102ec0 0x7f0e2c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e2c102500 0x7f0e2c197bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 --2- 172.21.15.73:0/408447680 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e2c101310 0x7f0e2c197690 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 >> 172.21.15.73:0/408447680 conn(0x7f0e2c0fc860 msgr2=0x7f0e2c105800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 shutdown_connections 2024-03-09T00:06:37.379 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.384+0000 7f0e334c0640 1 -- 172.21.15.73:0/408447680 wait complete. 2024-03-09T00:06:37.390 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:37.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 -- 172.21.15.73:0/3124801010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 msgr2=0x7ffbcc102a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 --2- 172.21.15.73:0/3124801010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc102a20 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7ffbc8009930 tx=0x7ffbc802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:37.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 -- 172.21.15.73:0/3124801010 shutdown_connections 2024-03-09T00:06:37.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 --2- 172.21.15.73:0/3124801010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 --2- 172.21.15.73:0/3124801010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc102a20 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 --2- 172.21.15.73:0/3124801010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 0x7ffbcc0ff9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 -- 172.21.15.73:0/3124801010 >> 172.21.15.73:0/3124801010 conn(0x7ffbcc0faf00 msgr2=0x7ffbcc0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 -- 172.21.15.73:0/3124801010 shutdown_connections 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.500+0000 7ffbd3969640 1 -- 172.21.15.73:0/3124801010 wait complete. 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 Processor -- start 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 -- start start 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 0x7ffbcc101f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc100a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc10d000 con 0x7ffbcc1025a0 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc10ce80 con 0x7ffbcc0ff5d0 2024-03-09T00:06:37.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd3969640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbcc10d180 con 0x7ffbcc102f60 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49432/0 (socket says 172.21.15.73:49432) 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 -- 172.21.15.73:0/811621557 learned_addr learned my addr 172.21.15.73:0/811621557 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd1705640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 0x7ffbcc101f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd1f06640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc100a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 -- 172.21.15.73:0/811621557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 msgr2=0x7ffbcc100a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc100a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 -- 172.21.15.73:0/811621557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 msgr2=0x7ffbcc101f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 0x7ffbcc101f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd0f04640 1 -- 172.21.15.73:0/811621557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbcc101320 con 0x7ffbcc1025a0 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.501+0000 7ffbd1f06640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbd0f04640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffbc802efd0 tx=0x7ffbc802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8041070 con 0x7ffbcc1025a0 2024-03-09T00:06:37.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffbc8004450 con 0x7ffbcc1025a0 2024-03-09T00:06:37.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbcc1a1730 con 0x7ffbcc1025a0 2024-03-09T00:06:37.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffbc8034440 con 0x7ffbcc1025a0 2024-03-09T00:06:37.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.502+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbcc1a1c80 con 0x7ffbcc1025a0 2024-03-09T00:06:37.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.503+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbcc1086d0 con 0x7ffbcc1025a0 2024-03-09T00:06:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.504+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ffbc8044070 con 0x7ffbcc1025a0 2024-03-09T00:06:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.505+0000 7ffbc27fc640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 0x7ffbb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.505+0000 7ffbd1705640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 0x7ffbb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.505+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ffbc80f0ed0 con 0x7ffbcc1025a0 2024-03-09T00:06:37.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.505+0000 7ffbd1705640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 0x7ffbb80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0696e0 tx=0x7ffbbc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.508+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffbc80bdbc0 con 0x7ffbcc1025a0 2024-03-09T00:06:37.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:37 smithi102 ceph-mon[36106]: pgmap v369: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:06:37.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.684+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ffbcc02d0e0 con 0x7ffbcc1025a0 2024-03-09T00:06:37.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.685+0000 7ffbc27fc640 1 -- 172.21.15.73:0/811621557 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7ffbc80bd4c0 con 0x7ffbcc1025a0 2024-03-09T00:06:37.679 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 msgr2=0x7ffbb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 0x7ffbb80a4cc0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffbcc0696e0 tx=0x7ffbbc005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 msgr2=0x7ffbcc100550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7ffbc802efd0 tx=0x7ffbc802c6c0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 shutdown_connections 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ffbb80a2800 0x7ffbb80a4cc0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbcc102f60 0x7ffbcc100a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbcc1025a0 0x7ffbcc100550 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 --2- 172.21.15.73:0/811621557 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbcc0ff5d0 0x7ffbcc101f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 >> 172.21.15.73:0/811621557 conn(0x7ffbcc0faf00 msgr2=0x7ffbcc1058a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 shutdown_connections 2024-03-09T00:06:37.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.687+0000 7ffbd3969640 1 -- 172.21.15.73:0/811621557 wait complete. 2024-03-09T00:06:37.692 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- 172.21.15.73:0/2861911215 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af41013f0 msgr2=0x7f5af4101870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 --2- 172.21.15.73:0/2861911215 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af41013f0 0x7f5af4101870 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7f5ae8009ca0 tx=0x7f5ae802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- 172.21.15.73:0/2861911215 shutdown_connections 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 --2- 172.21.15.73:0/2861911215 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af4101db0 0x7f5af4108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 --2- 172.21.15.73:0/2861911215 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af41013f0 0x7f5af4101870 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 --2- 172.21.15.73:0/2861911215 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af4100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- 172.21.15.73:0/2861911215 >> 172.21.15.73:0/2861911215 conn(0x7f5af40fb790 msgr2=0x7f5af40fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- 172.21.15.73:0/2861911215 shutdown_connections 2024-03-09T00:06:37.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- 172.21.15.73:0/2861911215 wait complete. 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 Processor -- start 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.803+0000 7f5af9774640 1 -- start start 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af406a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af41013f0 0x7f5af419c400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410bf70 con 0x7f5af4101db0 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410bdf0 con 0x7f5af4100200 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af410c0f0 con 0x7f5af41013f0 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49456/0 (socket says 172.21.15.73:49456) 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 -- 172.21.15.73:0/292386261 learned_addr learned my addr 172.21.15.73:0/292386261 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:37.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af2ffd640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af406a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 -- 172.21.15.73:0/292386261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af41013f0 msgr2=0x7f5af419c400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af27fc640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af41013f0 0x7f5af419c400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af41013f0 0x7f5af419c400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 -- 172.21.15.73:0/292386261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 msgr2=0x7f5af406a430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af406a430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 -- 172.21.15.73:0/292386261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af419f330 con 0x7f5af4101db0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af2ffd640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af406a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af37fe640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009fd0 tx=0x7f5ae400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae4011830 con 0x7f5af4101db0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af419f620 con 0x7f5af4101db0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ae4016070 con 0x7f5af4101db0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ae400f2e0 con 0x7f5af4101db0 2024-03-09T00:06:37.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.804+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af419fba0 con 0x7f5af4101db0 2024-03-09T00:06:37.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.805+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af41075f0 con 0x7f5af4101db0 2024-03-09T00:06:37.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.806+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5ae40119d0 con 0x7f5af4101db0 2024-03-09T00:06:37.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.807+0000 7f5adbfff640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 0x7f5adc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:37.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.807+0000 7f5af2ffd640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 0x7f5adc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:37.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.807+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5ae40ce590 con 0x7f5af4101db0 2024-03-09T00:06:37.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.807+0000 7f5af2ffd640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 0x7f5adc0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:37.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.811+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5ae409b280 con 0x7f5af4101db0 2024-03-09T00:06:37.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.991+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5af4061ef0 con 0x7f5af4101db0 2024-03-09T00:06:37.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.991+0000 7f5adbfff640 1 -- 172.21.15.73:0/292386261 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5ae409ab80 con 0x7f5af4101db0 2024-03-09T00:06:37.986 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 msgr2=0x7f5adc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 0x7f5adc0a4cc0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f5aec006fd0 tx=0x7f5aec008040 comp rx=0 tx=0).stop 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 msgr2=0x7f5af419ea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7f5ae4009fd0 tx=0x7f5ae400b8e0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 shutdown_connections 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5adc0a2800 0x7f5adc0a4cc0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5af4101db0 0x7f5af419ea70 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5af41013f0 0x7f5af419c400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 --2- 172.21.15.73:0/292386261 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5af4100200 0x7f5af406a430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:37.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 >> 172.21.15.73:0/292386261 conn(0x7f5af40fb790 msgr2=0x7f5af41047c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:37.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 shutdown_connections 2024-03-09T00:06:37.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:37.993+0000 7f5af9774640 1 -- 172.21.15.73:0/292386261 wait complete. 2024-03-09T00:06:37.999 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:38.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 -- 172.21.15.73:0/3954192074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 msgr2=0x7fe078104160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 --2- 172.21.15.73:0/3954192074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe078104160 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fe070009900 tx=0x7fe07002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:38.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 -- 172.21.15.73:0/3954192074 shutdown_connections 2024-03-09T00:06:38.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 --2- 172.21.15.73:0/3954192074 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 0x7fe078109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 --2- 172.21.15.73:0/3954192074 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 0x7fe078106a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 --2- 172.21.15.73:0/3954192074 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe078104160 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 -- 172.21.15.73:0/3954192074 >> 172.21.15.73:0/3954192074 conn(0x7fe0780fb780 msgr2=0x7fe0780fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 -- 172.21.15.73:0/3954192074 shutdown_connections 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.110+0000 7fe07cda9640 1 -- 172.21.15.73:0/3954192074 wait complete. 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 Processor -- start 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 -- start start 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 0x7fe078197bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 0x7fe07819ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07810cfe0 con 0x7fe078101d70 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07810ce60 con 0x7fe0781046a0 2024-03-09T00:06:38.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07cda9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe07810d160 con 0x7fe078106fd0 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49462/0 (socket says 172.21.15.73:49462) 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.111+0000 7fe07659c640 1 -- 172.21.15.73:0/2160650980 learned_addr learned my addr 172.21.15.73:0/2160650980 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe076d9d640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 0x7fe07819ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 -- 172.21.15.73:0/2160650980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 msgr2=0x7fe07819ec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 0x7fe07819ec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 -- 172.21.15.73:0/2160650980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 msgr2=0x7fe078197bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe075d9b640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 0x7fe078197bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 0x7fe078197bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 -- 172.21.15.73:0/2160650980 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07819f2d0 con 0x7fe078101d70 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07659c640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe070009230 tx=0x7fe070004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe070041070 con 0x7fe078101d70 2024-03-09T00:06:38.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe07819f560 con 0x7fe078101d70 2024-03-09T00:06:38.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0700054e0 con 0x7fe078101d70 2024-03-09T00:06:38.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0700346a0 con 0x7fe078101d70 2024-03-09T00:06:38.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.112+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe07819fa50 con 0x7fe078101d70 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.113+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780fd270 con 0x7fe078101d70 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.114+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe070044070 con 0x7fe078101d70 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.115+0000 7fe0677fe640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.115+0000 7fe075d9b640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.115+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe0700f0c50 con 0x7fe078101d70 2024-03-09T00:06:38.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.115+0000 7fe075d9b640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 0x7fe0600a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe078198bc0 tx=0x7fe06c008230 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.119+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe0700f6020 con 0x7fe078101d70 2024-03-09T00:06:38.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.293+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7fe078061ef0 con 0x7fe078101d70 2024-03-09T00:06:38.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.294+0000 7fe0677fe640 1 -- 172.21.15.73:0/2160650980 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7fe0700bd940 con 0x7fe078101d70 2024-03-09T00:06:38.289 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 msgr2=0x7fe0600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 0x7fe0600a4cc0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fe078198bc0 tx=0x7fe06c008230 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 msgr2=0x7fe0781976b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fe070009230 tx=0x7fe070004040 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 shutdown_connections 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe0600a2800 0x7fe0600a4cc0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe078106fd0 0x7fe07819ec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0781046a0 0x7fe078197bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 --2- 172.21.15.73:0/2160650980 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe078101d70 0x7fe0781976b0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 >> 172.21.15.73:0/2160650980 conn(0x7fe0780fb780 msgr2=0x7fe0781026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 shutdown_connections 2024-03-09T00:06:38.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.296+0000 7fe07cda9640 1 -- 172.21.15.73:0/2160650980 wait complete. 2024-03-09T00:06:38.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1670315745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:38.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/408447680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/811621557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:38 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/292386261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.302 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1670315745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/408447680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/811621557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:38 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/292386261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.409+0000 7f5e7d461640 1 -- 172.21.15.73:0/115744675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101dc0 msgr2=0x7f5e78108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.409+0000 7f5e7d461640 1 --2- 172.21.15.73:0/115744675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101dc0 0x7f5e78108650 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f5e70009ca0 tx=0x7f5e7002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- 172.21.15.73:0/115744675 shutdown_connections 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- 172.21.15.73:0/115744675 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101dc0 0x7f5e78108650 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- 172.21.15.73:0/115744675 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101400 0x7f5e78101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- 172.21.15.73:0/115744675 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e78100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- 172.21.15.73:0/115744675 >> 172.21.15.73:0/115744675 conn(0x7f5e780fb7a0 msgr2=0x7f5e780fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- 172.21.15.73:0/115744675 shutdown_connections 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- 172.21.15.73:0/115744675 wait complete. 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 Processor -- start 2024-03-09T00:06:38.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- start start 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e781975c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101dc0 0x7f5e7819eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7810bf00 con 0x7f5e78101400 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7810bd80 con 0x7f5e78101dc0 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.410+0000 7f5e7d461640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e7810c080 con 0x7f5e78100210 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49470/0 (socket says 172.21.15.73:49470) 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 -- 172.21.15.73:0/1482040940 learned_addr learned my addr 172.21.15.73:0/1482040940 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e76ffd640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e781975c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e777fe640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101dc0 0x7f5e7819eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 -- 172.21.15.73:0/1482040940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 msgr2=0x7f5e781975c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e781975c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 -- 172.21.15.73:0/1482040940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101dc0 msgr2=0x7f5e7819eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101dc0 0x7f5e7819eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 -- 172.21.15.73:0/1482040940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e7819f1e0 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e76ffd640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e781975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e6e7fc640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c920 tx=0x7f5e6000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60007c70 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5e60007e10 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e7819f4d0 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5e60018890 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.411+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e7819fa20 con 0x7f5e78101400 2024-03-09T00:06:38.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.412+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e7819fce0 con 0x7f5e78101400 2024-03-09T00:06:38.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.413+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5e60018a70 con 0x7f5e78101400 2024-03-09T00:06:38.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.414+0000 7f5e74ff9640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 0x7f5e640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.414+0000 7f5e76ffd640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 0x7f5e640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.414+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5e600cefe0 con 0x7f5e78101400 2024-03-09T00:06:38.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.414+0000 7f5e76ffd640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 0x7f5e640a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e78101260 tx=0x7f5e68008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.418+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5e60014030 con 0x7f5e78101400 2024-03-09T00:06:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1670315745' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/408447680' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/811621557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:38 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/292386261' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:38.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.593+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e7819feb0 con 0x7f5e78101400 2024-03-09T00:06:38.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.594+0000 7f5e74ff9640 1 -- 172.21.15.73:0/1482040940 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7f5e6000de20 con 0x7f5e78101400 2024-03-09T00:06:38.589 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:38.590 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 msgr2=0x7f5e640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 0x7f5e640a4cc0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f5e78101260 tx=0x7f5e68008040 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 msgr2=0x7f5e78197b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f5e6000c920 tx=0x7f5e6000cde0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 shutdown_connections 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5e640a2800 0x7f5e640a4cc0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5e78101dc0 0x7f5e7819eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5e78101400 0x7f5e78197b00 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 --2- 172.21.15.73:0/1482040940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5e78100210 0x7f5e781975c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 >> 172.21.15.73:0/1482040940 conn(0x7f5e780fb7a0 msgr2=0x7f5e780fd150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.596+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 shutdown_connections 2024-03-09T00:06:38.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.597+0000 7f5e7d461640 1 -- 172.21.15.73:0/1482040940 wait complete. 2024-03-09T00:06:38.603 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:38.604 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- 172.21.15.73:0/3443968347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 msgr2=0x7f40d8101730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- 172.21.15.73:0/3443968347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d8101730 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f40d0009ca0 tx=0x7f40d002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- 172.21.15.73:0/3443968347 shutdown_connections 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- 172.21.15.73:0/3443968347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40d8102ec0 0x7f40d8109750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- 172.21.15.73:0/3443968347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 0x7f40d8102980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- 172.21.15.73:0/3443968347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d8101730 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- 172.21.15.73:0/3443968347 >> 172.21.15.73:0/3443968347 conn(0x7f40d80fc880 msgr2=0x7f40d80fecc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- 172.21.15.73:0/3443968347 shutdown_connections 2024-03-09T00:06:38.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- 172.21.15.73:0/3443968347 wait complete. 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 Processor -- start 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 -- start start 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 0x7f40d8197b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.714+0000 7f40df36f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40d8102ec0 0x7f40d819eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40df36f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d810cfa0 con 0x7f40d8101310 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40df36f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d810ce20 con 0x7f40d8102500 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40df36f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40d810d120 con 0x7f40d8102ec0 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60096/0 (socket says 172.21.15.73:60096) 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 -- 172.21.15.73:0/1230375292 learned_addr learned my addr 172.21.15.73:0/1230375292 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dc90a640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 0x7f40d8197b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 -- 172.21.15.73:0/1230375292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40d8102ec0 msgr2=0x7f40d819eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40d8102ec0 0x7f40d819eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 -- 172.21.15.73:0/1230375292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 msgr2=0x7f40d8197b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 0x7f40d8197b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 -- 172.21.15.73:0/1230375292 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d819f210 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40dd10b640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000f80 tx=0x7f40d0032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d0039040 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40d0044070 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40d819f4a0 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40d00042a0 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.715+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40d819f9b0 con 0x7f40d8101310 2024-03-09T00:06:38.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.716+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40d81086b0 con 0x7f40d8101310 2024-03-09T00:06:38.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.717+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f40d0003970 con 0x7f40d8101310 2024-03-09T00:06:38.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.718+0000 7f40d67fc640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 0x7f40c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:38.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.718+0000 7f40dc90a640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 0x7f40c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:38.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.718+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f40d00f1270 con 0x7f40d8101310 2024-03-09T00:06:38.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.718+0000 7f40dc90a640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 0x7f40c00a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40d8198b00 tx=0x7f40cc00a4c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:38.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.722+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f40d003e3a0 con 0x7f40d8101310 2024-03-09T00:06:38.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.894+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f40d8061ef0 con 0x7f40d8101310 2024-03-09T00:06:38.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.894+0000 7f40d67fc640 1 -- 172.21.15.73:0/1230375292 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f40d00bdf60 con 0x7f40d8101310 2024-03-09T00:06:38.889 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:38.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 msgr2=0x7f40c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 0x7f40c00a4d40 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f40d8198b00 tx=0x7f40cc00a4c0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 msgr2=0x7f40d81975f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f40d0000f80 tx=0x7f40d0032ec0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 shutdown_connections 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f40c00a2880 0x7f40c00a4d40 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f40d8102ec0 0x7f40d819eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f40d8102500 0x7f40d8197b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 --2- 172.21.15.73:0/1230375292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f40d8101310 0x7f40d81975f0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:38.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 >> 172.21.15.73:0/1230375292 conn(0x7f40d80fc880 msgr2=0x7f40d8105880 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:38.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 shutdown_connections 2024-03-09T00:06:38.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:38.896+0000 7f40df36f640 1 -- 172.21.15.73:0/1230375292 wait complete. 2024-03-09T00:06:38.902 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:39.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 -- 172.21.15.73:0/3017441353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab4101da0 msgr2=0x7f5ab4108630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/3017441353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab4101da0 0x7f5ab4108630 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 crypto rx=0x7f5aac009ca0 tx=0x7f5aac02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 -- 172.21.15.73:0/3017441353 shutdown_connections 2024-03-09T00:06:39.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/3017441353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab4101da0 0x7f5ab4108630 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/3017441353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab41013e0 0x7f5ab4101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/3017441353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41001f0 0x7f5ab4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 -- 172.21.15.73:0/3017441353 >> 172.21.15.73:0/3017441353 conn(0x7f5ab40fb780 msgr2=0x7f5ab40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 -- 172.21.15.73:0/3017441353 shutdown_connections 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.012+0000 7f5ab94ab640 1 -- 172.21.15.73:0/3017441353 wait complete. 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 Processor -- start 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 -- start start 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41013e0 0x7f5ab41027c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4101da0 0x7f5ab4102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab410bec0 con 0x7f5ab41001f0 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab410bd40 con 0x7f5ab41013e0 2024-03-09T00:06:39.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab94ab640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ab410c040 con 0x7f5ab4101da0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60118/0 (socket says 172.21.15.73:60118) 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 -- 172.21.15.73:0/439267907 learned_addr learned my addr 172.21.15.73:0/439267907 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5aabfff640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41013e0 0x7f5ab41027c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab37fe640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4101da0 0x7f5ab4102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 -- 172.21.15.73:0/439267907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4101da0 msgr2=0x7f5ab4102d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4101da0 0x7f5ab4102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 -- 172.21.15.73:0/439267907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41013e0 msgr2=0x7f5ab41027c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41013e0 0x7f5ab41027c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 -- 172.21.15.73:0/439267907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab4103590 con 0x7f5ab41001f0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab2ffd640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400d770 tx=0x7f5aa400db70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.013+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4017070 con 0x7f5ab41001f0 2024-03-09T00:06:39.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.014+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ab41a1740 con 0x7f5ab41001f0 2024-03-09T00:06:39.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.014+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5aa4009d00 con 0x7f5ab41001f0 2024-03-09T00:06:39.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.014+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5aa4003d30 con 0x7f5ab41001f0 2024-03-09T00:06:39.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.014+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ab41a1c90 con 0x7f5ab41001f0 2024-03-09T00:06:39.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.014+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ab41a1f50 con 0x7f5ab41001f0 2024-03-09T00:06:39.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.015+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5aa4009820 con 0x7f5ab41001f0 2024-03-09T00:06:39.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.016+0000 7f5ab0ff9640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 0x7f5aa00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.016+0000 7f5aabfff640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 0x7f5aa00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.016+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5aa40ce790 con 0x7f5ab41001f0 2024-03-09T00:06:39.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.016+0000 7f5aabfff640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 0x7f5aa00a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4103ef0 tx=0x7f5a9c009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.020+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5aa409b480 con 0x7f5ab41001f0 2024-03-09T00:06:39.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.195+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5ab41a2120 con 0x7f5ab41001f0 2024-03-09T00:06:39.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.196+0000 7f5ab0ff9640 1 -- 172.21.15.73:0/439267907 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5aa409ad80 con 0x7f5ab41001f0 2024-03-09T00:06:39.191 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 msgr2=0x7f5aa00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 0x7f5aa00a4cc0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4103ef0 tx=0x7f5a9c009110 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 msgr2=0x7f5ab4104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7f5aa400d770 tx=0x7f5aa400db70 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 shutdown_connections 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5aa00a2800 0x7f5aa00a4cc0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5ab4101da0 0x7f5ab4102d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5ab41013e0 0x7f5ab41027c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 --2- 172.21.15.73:0/439267907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5ab41001f0 0x7f5ab4104180 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 >> 172.21.15.73:0/439267907 conn(0x7f5ab40fb780 msgr2=0x7f5ab40fd1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 shutdown_connections 2024-03-09T00:06:39.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.198+0000 7f5ab94ab640 1 -- 172.21.15.73:0/439267907 wait complete. 2024-03-09T00:06:39.202 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:39 smithi073 ceph-mon[32309]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:39.203 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2160650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.203 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1482040940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:39.203 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1230375292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.204 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:39.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.313+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/323195931 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 msgr2=0x7f5fd4101860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.313+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/323195931 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4101860 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009900 tx=0x7f5fd002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/323195931 shutdown_connections 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/323195931 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd4101da0 0x7f5fd4108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/323195931 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4101860 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/323195931 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fd41001f0 0x7f5fd4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/323195931 >> 172.21.15.73:0/323195931 conn(0x7f5fd40fb780 msgr2=0x7f5fd40fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/323195931 shutdown_connections 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/323195931 wait complete. 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 Processor -- start 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- start start 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd41001f0 0x7f5fd4197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fd4101da0 0x7f5fd419ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd410be80 con 0x7f5fd41013e0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd410bd00 con 0x7f5fd4101da0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fdb2c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fd410c000 con 0x7f5fd41001f0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fd885b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fd885b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60142/0 (socket says 172.21.15.73:60142) 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.314+0000 7f5fd885b640 1 -- 172.21.15.73:0/3798805855 learned_addr learned my addr 172.21.15.73:0/3798805855 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd905c640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd41001f0 0x7f5fd4197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 -- 172.21.15.73:0/3798805855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd41001f0 msgr2=0x7f5fd4197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd41001f0 0x7f5fd4197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 -- 172.21.15.73:0/3798805855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fd4101da0 msgr2=0x7f5fd419ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fd4101da0 0x7f5fd419ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 -- 172.21.15.73:0/3798805855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd419f240 con 0x7f5fd41013e0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fd885b640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009230 tx=0x7f5fd0005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0041070 con 0x7f5fd41013e0 2024-03-09T00:06:39.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fd00054e0 con 0x7f5fd41013e0 2024-03-09T00:06:39.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fd419f4d0 con 0x7f5fd41013e0 2024-03-09T00:06:39.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fd0034440 con 0x7f5fd41013e0 2024-03-09T00:06:39.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.315+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fd419fa10 con 0x7f5fd41013e0 2024-03-09T00:06:39.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.316+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fd419fc70 con 0x7f5fd41013e0 2024-03-09T00:06:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.317+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5fd0044070 con 0x7f5fd41013e0 2024-03-09T00:06:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.317+0000 7f5fca7fc640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 0x7f5fc00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.318+0000 7f5fd905c640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 0x7f5fc00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.318+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5fd00f0b50 con 0x7f5fd41013e0 2024-03-09T00:06:39.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.318+0000 7f5fd905c640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 0x7f5fc00a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4101240 tx=0x7f5fc4005c60 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.322+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5fd00bd840 con 0x7f5fd41013e0 2024-03-09T00:06:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:39 smithi179 ceph-mon[36637]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2160650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1482040940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:39.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1230375292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.497+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5fd419fe40 con 0x7f5fd41013e0 2024-03-09T00:06:39.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.497+0000 7f5fca7fc640 1 -- 172.21.15.73:0/3798805855 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f5fd00bd140 con 0x7f5fd41013e0 2024-03-09T00:06:39.492 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 msgr2=0x7f5fc00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 0x7f5fc00a4d40 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f5fd4101240 tx=0x7f5fc4005c60 comp rx=0 tx=0).stop 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 msgr2=0x7f5fd4197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009230 tx=0x7f5fd0005dd0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 shutdown_connections 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5fc00a2880 0x7f5fc00a4d40 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5fd4101da0 0x7f5fd419ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5fd41013e0 0x7f5fd4197b60 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 --2- 172.21.15.73:0/3798805855 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5fd41001f0 0x7f5fd4197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 >> 172.21.15.73:0/3798805855 conn(0x7f5fd40fb780 msgr2=0x7f5fd4068450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 shutdown_connections 2024-03-09T00:06:39.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.499+0000 7f5fdb2c0640 1 -- 172.21.15.73:0/3798805855 wait complete. 2024-03-09T00:06:39.505 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:39 smithi102 ceph-mon[36106]: pgmap v370: 129 pgs: 129 active+clean; 640 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:06:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2160650980' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1482040940' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:39.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1230375292' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 -- 172.21.15.73:0/3621908878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 msgr2=0x7f293c108640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 --2- 172.21.15.73:0/3621908878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c108640 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7f2924009900 tx=0x7f292402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 -- 172.21.15.73:0/3621908878 shutdown_connections 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 --2- 172.21.15.73:0/3621908878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c108640 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 --2- 172.21.15.73:0/3621908878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c1013f0 0x7f293c101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 --2- 172.21.15.73:0/3621908878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c100200 0x7f293c100620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 -- 172.21.15.73:0/3621908878 >> 172.21.15.73:0/3621908878 conn(0x7f293c0fb790 msgr2=0x7f293c0fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 -- 172.21.15.73:0/3621908878 shutdown_connections 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.615+0000 7f2942df4640 1 -- 172.21.15.73:0/3621908878 wait complete. 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 Processor -- start 2024-03-09T00:06:39.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 -- start start 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c100200 0x7f293c1964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c1013f0 0x7f293c196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c10be90 con 0x7f293c101db0 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c10bd10 con 0x7f293c1013f0 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2942df4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f293c10c010 con 0x7f293c100200 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60162/0 (socket says 172.21.15.73:60162) 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 -- 172.21.15.73:0/929869091 learned_addr learned my addr 172.21.15.73:0/929869091 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f293bfff640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c1013f0 0x7f293c196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 -- 172.21.15.73:0/929869091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c100200 msgr2=0x7f293c1964d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c100200 0x7f293c1964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 -- 172.21.15.73:0/929869091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c1013f0 msgr2=0x7f293c196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c1013f0 0x7f293c196a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 -- 172.21.15.73:0/929869091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f293c19e0f0 con 0x7f293c101db0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2941391640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2924009230 tx=0x7f2924004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.616+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2924041070 con 0x7f293c101db0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.617+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29240054e0 con 0x7f293c101db0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.617+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f293c19e380 con 0x7f293c101db0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.617+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29240346a0 con 0x7f293c101db0 2024-03-09T00:06:39.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.617+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f293c19e870 con 0x7f293c101db0 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.618+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f293c1074c0 con 0x7f293c101db0 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.619+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f2924044070 con 0x7f293c101db0 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.619+0000 7f2939ffb640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 0x7f29280a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.619+0000 7f2940b90640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 0x7f29280a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.619+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f29240f1130 con 0x7f293c101db0 2024-03-09T00:06:39.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.620+0000 7f2940b90640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 0x7f29280a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f293c101250 tx=0x7f29340071e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.623+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f29240bde20 con 0x7f293c101db0 2024-03-09T00:06:39.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.800+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f293c02ce70 con 0x7f293c101db0 2024-03-09T00:06:39.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.801+0000 7f2939ffb640 1 -- 172.21.15.73:0/929869091 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f29240bd720 con 0x7f293c101db0 2024-03-09T00:06:39.795 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:39.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.802+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 msgr2=0x7f29280a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.802+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 0x7f29280a4cc0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f293c101250 tx=0x7f29340071e0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.802+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 msgr2=0x7f293c19da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.802+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f2924009230 tx=0x7f2924004040 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 shutdown_connections 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f29280a2800 0x7f29280a4cc0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f293c101db0 0x7f293c19da80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f293c1013f0 0x7f293c196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 --2- 172.21.15.73:0/929869091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f293c100200 0x7f293c1964d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 >> 172.21.15.73:0/929869091 conn(0x7f293c0fb790 msgr2=0x7f293c104690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 shutdown_connections 2024-03-09T00:06:39.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.803+0000 7f2942df4640 1 -- 172.21.15.73:0/929869091 wait complete. 2024-03-09T00:06:39.808 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-09T00:06:39.809 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:39.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 -- 172.21.15.73:0/3034554857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff730101300 msgr2=0x7ff730101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 --2- 172.21.15.73:0/3034554857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff730101300 0x7ff730101720 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7ff72c009900 tx=0x7ff72c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 -- 172.21.15.73:0/3034554857 shutdown_connections 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 --2- 172.21.15.73:0/3034554857 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff730109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 --2- 172.21.15.73:0/3034554857 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff7301024f0 0x7ff730102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 --2- 172.21.15.73:0/3034554857 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff730101300 0x7ff730101720 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 -- 172.21.15.73:0/3034554857 >> 172.21.15.73:0/3034554857 conn(0x7ff7300fc8d0 msgr2=0x7ff7300fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 -- 172.21.15.73:0/3034554857 shutdown_connections 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.917+0000 7ff73800b640 1 -- 172.21.15.73:0/3034554857 wait complete. 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 Processor -- start 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 -- start start 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 0x7ff730197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff73019ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010cf90 con 0x7ff7301024f0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010ce10 con 0x7ff730101300 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff73800b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010d110 con 0x7ff730102eb0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60180/0 (socket says 172.21.15.73:60180) 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 -- 172.21.15.73:0/1998829124 learned_addr learned my addr 172.21.15.73:0/1998829124 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff735da7640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 0x7ff730197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7365a8640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff73019ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 -- 172.21.15.73:0/1998829124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 msgr2=0x7ff73019ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff73019ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 -- 172.21.15.73:0/1998829124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 msgr2=0x7ff730197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 0x7ff730197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 -- 172.21.15.73:0/1998829124 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73019f230 con 0x7ff7301024f0 2024-03-09T00:06:39.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7365a8640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff73019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff735da7640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 0x7ff730197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.918+0000 7ff7355a6640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff72000d950 tx=0x7ff72000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720017030 con 0x7ff7301024f0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff73019f520 con 0x7ff7301024f0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff720004020 con 0x7ff7301024f0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7200047a0 con 0x7ff7301024f0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff73019fa70 con 0x7ff7301024f0 2024-03-09T00:06:39.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.919+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7301086a0 con 0x7ff7301024f0 2024-03-09T00:06:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.921+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff7200041d0 con 0x7ff7301024f0 2024-03-09T00:06:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.921+0000 7ff726ffd640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 0x7ff71c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.921+0000 7ff735da7640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 0x7ff71c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.921+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff7200cf020 con 0x7ff7301024f0 2024-03-09T00:06:39.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.922+0000 7ff735da7640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 0x7ff71c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff72c02c7b0 tx=0x7ff72c005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:39.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:39.925+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff72009bd10 con 0x7ff7301024f0 2024-03-09T00:06:40.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.141+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff73002d0e0 con 0x7ff7301024f0 2024-03-09T00:06:40.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.141+0000 7ff726ffd640 1 -- 172.21.15.73:0/1998829124 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7ff72009b610 con 0x7ff7301024f0 2024-03-09T00:06:40.136 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:40.136 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:40.137 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 msgr2=0x7ff71c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 0x7ff71c0a4cc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7ff72c02c7b0 tx=0x7ff72c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 msgr2=0x7ff730197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7ff72000d950 tx=0x7ff72000de10 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 shutdown_connections 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff71c0a2800 0x7ff71c0a4cc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff730102eb0 0x7ff73019ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff7301024f0 0x7ff730197b50 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 --2- 172.21.15.73:0/1998829124 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff730101300 0x7ff730197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 >> 172.21.15.73:0/1998829124 conn(0x7ff7300fc8d0 msgr2=0x7ff730105870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:40.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 shutdown_connections 2024-03-09T00:06:40.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.144+0000 7ff73800b640 1 -- 172.21.15.73:0/1998829124 wait complete. 2024-03-09T00:06:40.148 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:40 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/439267907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.148 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:40 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3798805855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.148 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:40 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/929869091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.150 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-09T00:06:40.150 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-09T00:06:40.150 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T00:06:40.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 -- 172.21.15.73:0/607376318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8101dc0 msgr2=0x7ff5a8108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 --2- 172.21.15.73:0/607376318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8101dc0 0x7ff5a8108650 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7ff5a0009ca0 tx=0x7ff5a002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 -- 172.21.15.73:0/607376318 shutdown_connections 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 --2- 172.21.15.73:0/607376318 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8101dc0 0x7ff5a8108650 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 --2- 172.21.15.73:0/607376318 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 0x7ff5a8101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.256+0000 7ff5adedc640 1 --2- 172.21.15.73:0/607376318 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8100210 0x7ff5a8100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- 172.21.15.73:0/607376318 >> 172.21.15.73:0/607376318 conn(0x7ff5a80fb780 msgr2=0x7ff5a80fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- 172.21.15.73:0/607376318 shutdown_connections 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- 172.21.15.73:0/607376318 wait complete. 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 Processor -- start 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- start start 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 0x7ff5a8196a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 0x7ff5a819da80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a810bea0 con 0x7ff5a8100210 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a810bd20 con 0x7ff5a8101dc0 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5adedc640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5a810c020 con 0x7ff5a8101400 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.257+0000 7ff5a77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60192/0 (socket says 172.21.15.73:60192) 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 -- 172.21.15.73:0/1932670296 learned_addr learned my addr 172.21.15.73:0/1932670296 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a7fff640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 0x7ff5a819da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff59effd640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 0x7ff5a8196a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 -- 172.21.15.73:0/1932670296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 msgr2=0x7ff5a8196a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 0x7ff5a8196a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 -- 172.21.15.73:0/1932670296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 msgr2=0x7ff5a819da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 0x7ff5a819da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 -- 172.21.15.73:0/1932670296 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5a819e0f0 con 0x7ff5a8100210 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a7fff640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 0x7ff5a819da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a77fe640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d950 tx=0x7ff59800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:40.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff598017020 con 0x7ff5a8100210 2024-03-09T00:06:40.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff598009c30 con 0x7ff5a8100210 2024-03-09T00:06:40.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5a819e3e0 con 0x7ff5a8100210 2024-03-09T00:06:40.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff59800f720 con 0x7ff5a8100210 2024-03-09T00:06:40.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.258+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5a819e930 con 0x7ff5a8100210 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.259+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff5a8107530 con 0x7ff5a8100210 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.260+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff59801a020 con 0x7ff5a8100210 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.261+0000 7ff5a57fa640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 0x7ff5880a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.261+0000 7ff59effd640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 0x7ff5880a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.261+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff5980cedf0 con 0x7ff5a8100210 2024-03-09T00:06:40.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.261+0000 7ff59effd640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 0x7ff5880a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff590005d80 tx=0x7ff590005c90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:40.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.265+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff598021020 con 0x7ff5a8100210 2024-03-09T00:06:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:40 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/439267907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:40 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3798805855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:40 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/929869091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.436+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff5a8061730 con 0x7ff5880a2800 2024-03-09T00:06:40.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.445+0000 7ff5a57fa640 1 -- 172.21.15.73:0/1932670296 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340392 (secure 0 0 0) 0x7ff5a8061730 con 0x7ff5880a2800 2024-03-09T00:06:40.444 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:40.446 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:06:40.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 msgr2=0x7ff5880a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 0x7ff5880a4cc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7ff590005d80 tx=0x7ff590005c90 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 msgr2=0x7ff5a81964d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7ff59800d950 tx=0x7ff59800de10 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 shutdown_connections 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff5880a2800 0x7ff5880a4cc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5a8101dc0 0x7ff5a819da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5a8101400 0x7ff5a8196a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 --2- 172.21.15.73:0/1932670296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5a8100210 0x7ff5a81964d0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 >> 172.21.15.73:0/1932670296 conn(0x7ff5a80fb780 msgr2=0x7ff5a8104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 shutdown_connections 2024-03-09T00:06:40.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.449+0000 7ff5adedc640 1 -- 172.21.15.73:0/1932670296 wait complete. 2024-03-09T00:06:40.455 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-09T00:06:39.967346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660686,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":394,"num_read_kb":319,"num_write":272,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":715,"ondisk_log_size":715,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396832,"kb_used_data":14340,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544992,"statfs":{"total":1151940427776,"available":1151534071808,"internally_reserved":0,"allocated":14684160,"data_stored":8183238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079857+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:06:06.079857+0000","last_peered":"2024-03-09T00:06:06.079857+0000","last_clean":"2024-03-09T00:06:06.079857+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:06:06.079857+0000","last_undegraded":"2024-03-09T00:06:06.079857+0000","last_fullsized":"2024-03-09T00:06:06.079857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081264+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:06:06.081264+0000","last_peered":"2024-03-09T00:06:06.081264+0000","last_clean":"2024-03-09T00:06:06.081264+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:06:06.081264+0000","last_undegraded":"2024-03-09T00:06:06.081264+0000","last_fullsized":"2024-03-09T00:06:06.081264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081167+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:06:06.081167+0000","last_peered":"2024-03-09T00:06:06.081167+0000","last_clean":"2024-03-09T00:06:06.081167+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:06:06.081167+0000","last_undegraded":"2024-03-09T00:06:06.081167+0000","last_fullsized":"2024-03-09T00:06:06.081167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272147+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:06:06.272147+0000","last_peered":"2024-03-09T00:06:06.272147+0000","last_clean":"2024-03-09T00:06:06.272147+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:06:06.272147+0000","last_undegraded":"2024-03-09T00:06:06.272147+0000","last_fullsized":"2024-03-09T00:06:06.272147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081462+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:06:06.081462+0000","last_peered":"2024-03-09T00:06:06.081462+0000","last_clean":"2024-03-09T00:06:06.081462+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:06:06.081462+0000","last_undegraded":"2024-03-09T00:06:06.081462+0000","last_fullsized":"2024-03-09T00:06:06.081462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075196+0000","last_change":"2024-03-09T00:05:05.058033+0000","last_active":"2024-03-09T00:06:06.075196+0000","last_peered":"2024-03-09T00:06:06.075196+0000","last_clean":"2024-03-09T00:06:06.075196+0000","last_became_active":"2024-03-09T00:05:05.057575+0000","last_became_peered":"2024-03-09T00:05:05.057575+0000","last_unstale":"2024-03-09T00:06:06.075196+0000","last_undegraded":"2024-03-09T00:06:06.075196+0000","last_fullsized":"2024-03-09T00:06:06.075196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:35:58.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079431+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:06:06.079431+0000","last_peered":"2024-03-09T00:06:06.079431+0000","last_clean":"2024-03-09T00:06:06.079431+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:06:06.079431+0000","last_undegraded":"2024-03-09T00:06:06.079431+0000","last_fullsized":"2024-03-09T00:06:06.079431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921426+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:06:06.921426+0000","last_peered":"2024-03-09T00:06:06.921426+0000","last_clean":"2024-03-09T00:06:06.921426+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:06:06.921426+0000","last_undegraded":"2024-03-09T00:06:06.921426+0000","last_fullsized":"2024-03-09T00:06:06.921426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513955+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:06:06.513955+0000","last_peered":"2024-03-09T00:06:06.513955+0000","last_clean":"2024-03-09T00:06:06.513955+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:06:06.513955+0000","last_undegraded":"2024-03-09T00:06:06.513955+0000","last_fullsized":"2024-03-09T00:06:06.513955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075525+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:06:06.075525+0000","last_peered":"2024-03-09T00:06:06.075525+0000","last_clean":"2024-03-09T00:06:06.075525+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:06:06.075525+0000","last_undegraded":"2024-03-09T00:06:06.075525+0000","last_fullsized":"2024-03-09T00:06:06.075525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079896+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:06:06.079896+0000","last_peered":"2024-03-09T00:06:06.079896+0000","last_clean":"2024-03-09T00:06:06.079896+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:06:06.079896+0000","last_undegraded":"2024-03-09T00:06:06.079896+0000","last_fullsized":"2024-03-09T00:06:06.079896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271903+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:06:06.271903+0000","last_peered":"2024-03-09T00:06:06.271903+0000","last_clean":"2024-03-09T00:06:06.271903+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:06:06.271903+0000","last_undegraded":"2024-03-09T00:06:06.271903+0000","last_fullsized":"2024-03-09T00:06:06.271903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076426+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:06:06.076426+0000","last_peered":"2024-03-09T00:06:06.076426+0000","last_clean":"2024-03-09T00:06:06.076426+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:06:06.076426+0000","last_undegraded":"2024-03-09T00:06:06.076426+0000","last_fullsized":"2024-03-09T00:06:06.076426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081033+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:06:06.081033+0000","last_peered":"2024-03-09T00:06:06.081033+0000","last_clean":"2024-03-09T00:06:06.081033+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:06:06.081033+0000","last_undegraded":"2024-03-09T00:06:06.081033+0000","last_fullsized":"2024-03-09T00:06:06.081033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080896+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:06:06.080896+0000","last_peered":"2024-03-09T00:06:06.080896+0000","last_clean":"2024-03-09T00:06:06.080896+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:06:06.080896+0000","last_undegraded":"2024-03-09T00:06:06.080896+0000","last_fullsized":"2024-03-09T00:06:06.080896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513424+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:06:06.513424+0000","last_peered":"2024-03-09T00:06:06.513424+0000","last_clean":"2024-03-09T00:06:06.513424+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:06:06.513424+0000","last_undegraded":"2024-03-09T00:06:06.513424+0000","last_fullsized":"2024-03-09T00:06:06.513424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079620+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:06:06.079620+0000","last_peered":"2024-03-09T00:06:06.079620+0000","last_clean":"2024-03-09T00:06:06.079620+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:06:06.079620+0000","last_undegraded":"2024-03-09T00:06:06.079620+0000","last_fullsized":"2024-03-09T00:06:06.079620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277255+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:06:06.277255+0000","last_peered":"2024-03-09T00:06:06.277255+0000","last_clean":"2024-03-09T00:06:06.277255+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:06:06.277255+0000","last_undegraded":"2024-03-09T00:06:06.277255+0000","last_fullsized":"2024-03-09T00:06:06.277255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081723+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:06:06.081723+0000","last_peered":"2024-03-09T00:06:06.081723+0000","last_clean":"2024-03-09T00:06:06.081723+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:06:06.081723+0000","last_undegraded":"2024-03-09T00:06:06.081723+0000","last_fullsized":"2024-03-09T00:06:06.081723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081285+0000","last_change":"2024-03-09T00:05:05.138220+0000","last_active":"2024-03-09T00:06:06.081285+0000","last_peered":"2024-03-09T00:06:06.081285+0000","last_clean":"2024-03-09T00:06:06.081285+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.081285+0000","last_undegraded":"2024-03-09T00:06:06.081285+0000","last_fullsized":"2024-03-09T00:06:06.081285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:41.455601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277194+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:06:06.277194+0000","last_peered":"2024-03-09T00:06:06.277194+0000","last_clean":"2024-03-09T00:06:06.277194+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:06:06.277194+0000","last_undegraded":"2024-03-09T00:06:06.277194+0000","last_fullsized":"2024-03-09T00:06:06.277194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922002+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:06:06.922002+0000","last_peered":"2024-03-09T00:06:06.922002+0000","last_clean":"2024-03-09T00:06:06.922002+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.922002+0000","last_undegraded":"2024-03-09T00:06:06.922002+0000","last_fullsized":"2024-03-09T00:06:06.922002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271466+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:06:06.271466+0000","last_peered":"2024-03-09T00:06:06.271466+0000","last_clean":"2024-03-09T00:06:06.271466+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:06:06.271466+0000","last_undegraded":"2024-03-09T00:06:06.271466+0000","last_fullsized":"2024-03-09T00:06:06.271466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922049+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:06:06.922049+0000","last_peered":"2024-03-09T00:06:06.922049+0000","last_clean":"2024-03-09T00:06:06.922049+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:06:06.922049+0000","last_undegraded":"2024-03-09T00:06:06.922049+0000","last_fullsized":"2024-03-09T00:06:06.922049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081306+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:06:06.081306+0000","last_peered":"2024-03-09T00:06:06.081306+0000","last_clean":"2024-03-09T00:06:06.081306+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:06:06.081306+0000","last_undegraded":"2024-03-09T00:06:06.081306+0000","last_fullsized":"2024-03-09T00:06:06.081306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271674+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:06:06.271674+0000","last_peered":"2024-03-09T00:06:06.271674+0000","last_clean":"2024-03-09T00:06:06.271674+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:06:06.271674+0000","last_undegraded":"2024-03-09T00:06:06.271674+0000","last_fullsized":"2024-03-09T00:06:06.271674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082791+0000","last_change":"2024-03-09T00:05:05.142560+0000","last_active":"2024-03-09T00:06:06.082791+0000","last_peered":"2024-03-09T00:06:06.082791+0000","last_clean":"2024-03-09T00:06:06.082791+0000","last_became_active":"2024-03-09T00:05:05.139877+0000","last_became_peered":"2024-03-09T00:05:05.139877+0000","last_unstale":"2024-03-09T00:06:06.082791+0000","last_undegraded":"2024-03-09T00:06:06.082791+0000","last_fullsized":"2024-03-09T00:06:06.082791+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:30.441723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513607+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:06:06.513607+0000","last_peered":"2024-03-09T00:06:06.513607+0000","last_clean":"2024-03-09T00:06:06.513607+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:06:06.513607+0000","last_undegraded":"2024-03-09T00:06:06.513607+0000","last_fullsized":"2024-03-09T00:06:06.513607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271559+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:06:06.271559+0000","last_peered":"2024-03-09T00:06:06.271559+0000","last_clean":"2024-03-09T00:06:06.271559+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:06:06.271559+0000","last_undegraded":"2024-03-09T00:06:06.271559+0000","last_fullsized":"2024-03-09T00:06:06.271559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082413+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:06:06.082413+0000","last_peered":"2024-03-09T00:06:06.082413+0000","last_clean":"2024-03-09T00:06:06.082413+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:06:06.082413+0000","last_undegraded":"2024-03-09T00:06:06.082413+0000","last_fullsized":"2024-03-09T00:06:06.082413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076341+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:06:06.076341+0000","last_peered":"2024-03-09T00:06:06.076341+0000","last_clean":"2024-03-09T00:06:06.076341+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:06:06.076341+0000","last_undegraded":"2024-03-09T00:06:06.076341+0000","last_fullsized":"2024-03-09T00:06:06.076341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081523+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:06:06.081523+0000","last_peered":"2024-03-09T00:06:06.081523+0000","last_clean":"2024-03-09T00:06:06.081523+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:06:06.081523+0000","last_undegraded":"2024-03-09T00:06:06.081523+0000","last_fullsized":"2024-03-09T00:06:06.081523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075947+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:06:06.075947+0000","last_peered":"2024-03-09T00:06:06.075947+0000","last_clean":"2024-03-09T00:06:06.075947+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:06:06.075947+0000","last_undegraded":"2024-03-09T00:06:06.075947+0000","last_fullsized":"2024-03-09T00:06:06.075947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082037+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:06:06.082037+0000","last_peered":"2024-03-09T00:06:06.082037+0000","last_clean":"2024-03-09T00:06:06.082037+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:06:06.082037+0000","last_undegraded":"2024-03-09T00:06:06.082037+0000","last_fullsized":"2024-03-09T00:06:06.082037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081531+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:06:06.081531+0000","last_peered":"2024-03-09T00:06:06.081531+0000","last_clean":"2024-03-09T00:06:06.081531+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:06:06.081531+0000","last_undegraded":"2024-03-09T00:06:06.081531+0000","last_fullsized":"2024-03-09T00:06:06.081531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075185+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:06:06.075185+0000","last_peered":"2024-03-09T00:06:06.075185+0000","last_clean":"2024-03-09T00:06:06.075185+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:06:06.075185+0000","last_undegraded":"2024-03-09T00:06:06.075185+0000","last_fullsized":"2024-03-09T00:06:06.075185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076123+0000","last_change":"2024-03-09T00:05:05.056415+0000","last_active":"2024-03-09T00:06:06.076123+0000","last_peered":"2024-03-09T00:06:06.076123+0000","last_clean":"2024-03-09T00:06:06.076123+0000","last_became_active":"2024-03-09T00:05:05.056025+0000","last_became_peered":"2024-03-09T00:05:05.056025+0000","last_unstale":"2024-03-09T00:06:06.076123+0000","last_undegraded":"2024-03-09T00:06:06.076123+0000","last_fullsized":"2024-03-09T00:06:06.076123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:29.970623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079682+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:06:06.079682+0000","last_peered":"2024-03-09T00:06:06.079682+0000","last_clean":"2024-03-09T00:06:06.079682+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:06:06.079682+0000","last_undegraded":"2024-03-09T00:06:06.079682+0000","last_fullsized":"2024-03-09T00:06:06.079682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076321+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:06:06.076321+0000","last_peered":"2024-03-09T00:06:06.076321+0000","last_clean":"2024-03-09T00:06:06.076321+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:06:06.076321+0000","last_undegraded":"2024-03-09T00:06:06.076321+0000","last_fullsized":"2024-03-09T00:06:06.076321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921566+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:06:06.921566+0000","last_peered":"2024-03-09T00:06:06.921566+0000","last_clean":"2024-03-09T00:06:06.921566+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:06:06.921566+0000","last_undegraded":"2024-03-09T00:06:06.921566+0000","last_fullsized":"2024-03-09T00:06:06.921566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081208+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:06:06.081208+0000","last_peered":"2024-03-09T00:06:06.081208+0000","last_clean":"2024-03-09T00:06:06.081208+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:06:06.081208+0000","last_undegraded":"2024-03-09T00:06:06.081208+0000","last_fullsized":"2024-03-09T00:06:06.081208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075906+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:06:06.075906+0000","last_peered":"2024-03-09T00:06:06.075906+0000","last_clean":"2024-03-09T00:06:06.075906+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:06:06.075906+0000","last_undegraded":"2024-03-09T00:06:06.075906+0000","last_fullsized":"2024-03-09T00:06:06.075906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276794+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:06:06.276794+0000","last_peered":"2024-03-09T00:06:06.276794+0000","last_clean":"2024-03-09T00:06:06.276794+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:06:06.276794+0000","last_undegraded":"2024-03-09T00:06:06.276794+0000","last_fullsized":"2024-03-09T00:06:06.276794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921459+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:06:06.921459+0000","last_peered":"2024-03-09T00:06:06.921459+0000","last_clean":"2024-03-09T00:06:06.921459+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:06:06.921459+0000","last_undegraded":"2024-03-09T00:06:06.921459+0000","last_fullsized":"2024-03-09T00:06:06.921459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075432+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:06:06.075432+0000","last_peered":"2024-03-09T00:06:06.075432+0000","last_clean":"2024-03-09T00:06:06.075432+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:06:06.075432+0000","last_undegraded":"2024-03-09T00:06:06.075432+0000","last_fullsized":"2024-03-09T00:06:06.075432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513829+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:06:06.513829+0000","last_peered":"2024-03-09T00:06:06.513829+0000","last_clean":"2024-03-09T00:06:06.513829+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:06:06.513829+0000","last_undegraded":"2024-03-09T00:06:06.513829+0000","last_fullsized":"2024-03-09T00:06:06.513829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081694+0000","last_change":"2024-03-09T00:05:05.139426+0000","last_active":"2024-03-09T00:06:06.081694+0000","last_peered":"2024-03-09T00:06:06.081694+0000","last_clean":"2024-03-09T00:06:06.081694+0000","last_became_active":"2024-03-09T00:05:05.138971+0000","last_became_peered":"2024-03-09T00:05:05.138971+0000","last_unstale":"2024-03-09T00:06:06.081694+0000","last_undegraded":"2024-03-09T00:06:06.081694+0000","last_fullsized":"2024-03-09T00:06:06.081694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:40:59.870972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.514952+0000","last_change":"2024-03-09T00:06:06.514952+0000","last_active":"2024-03-09T00:06:06.514952+0000","last_peered":"2024-03-09T00:06:06.514952+0000","last_clean":"2024-03-09T00:06:06.514952+0000","last_became_active":"2024-03-09T00:06:06.514480+0000","last_became_peered":"2024-03-09T00:06:06.514480+0000","last_unstale":"2024-03-09T00:06:06.514952+0000","last_undegraded":"2024-03-09T00:06:06.514952+0000","last_fullsized":"2024-03-09T00:06:06.514952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:03.976364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081301+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:06:06.081301+0000","last_peered":"2024-03-09T00:06:06.081301+0000","last_clean":"2024-03-09T00:06:06.081301+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:06:06.081301+0000","last_undegraded":"2024-03-09T00:06:06.081301+0000","last_fullsized":"2024-03-09T00:06:06.081301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271940+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:06:06.271940+0000","last_peered":"2024-03-09T00:06:06.271940+0000","last_clean":"2024-03-09T00:06:06.271940+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:06:06.271940+0000","last_undegraded":"2024-03-09T00:06:06.271940+0000","last_fullsized":"2024-03-09T00:06:06.271940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076324+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:06:06.076324+0000","last_peered":"2024-03-09T00:06:06.076324+0000","last_clean":"2024-03-09T00:06:06.076324+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:06:06.076324+0000","last_undegraded":"2024-03-09T00:06:06.076324+0000","last_fullsized":"2024-03-09T00:06:06.076324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513828+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:06:06.513828+0000","last_peered":"2024-03-09T00:06:06.513828+0000","last_clean":"2024-03-09T00:06:06.513828+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:06:06.513828+0000","last_undegraded":"2024-03-09T00:06:06.513828+0000","last_fullsized":"2024-03-09T00:06:06.513828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.074900+0000","last_change":"2024-03-09T00:05:05.058517+0000","last_active":"2024-03-09T00:06:06.074900+0000","last_peered":"2024-03-09T00:06:06.074900+0000","last_clean":"2024-03-09T00:06:06.074900+0000","last_became_active":"2024-03-09T00:05:05.057877+0000","last_became_peered":"2024-03-09T00:05:05.057877+0000","last_unstale":"2024-03-09T00:06:06.074900+0000","last_undegraded":"2024-03-09T00:06:06.074900+0000","last_fullsized":"2024-03-09T00:06:06.074900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:42:49.424790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'36","reported_seq":320,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:39.198178+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:06:39.198178+0000","last_peered":"2024-03-09T00:06:39.198178+0000","last_clean":"2024-03-09T00:06:39.198178+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:06:39.198178+0000","last_undegraded":"2024-03-09T00:06:39.198178+0000","last_fullsized":"2024-03-09T00:06:39.198178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":36,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082439+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:06:06.082439+0000","last_peered":"2024-03-09T00:06:06.082439+0000","last_clean":"2024-03-09T00:06:06.082439+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:06:06.082439+0000","last_undegraded":"2024-03-09T00:06:06.082439+0000","last_fullsized":"2024-03-09T00:06:06.082439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076594+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:06:06.076594+0000","last_peered":"2024-03-09T00:06:06.076594+0000","last_clean":"2024-03-09T00:06:06.076594+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:06:06.076594+0000","last_undegraded":"2024-03-09T00:06:06.076594+0000","last_fullsized":"2024-03-09T00:06:06.076594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081920+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:06:06.081920+0000","last_peered":"2024-03-09T00:06:06.081920+0000","last_clean":"2024-03-09T00:06:06.081920+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:06:06.081920+0000","last_undegraded":"2024-03-09T00:06:06.081920+0000","last_fullsized":"2024-03-09T00:06:06.081920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076436+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:06:06.076436+0000","last_peered":"2024-03-09T00:06:06.076436+0000","last_clean":"2024-03-09T00:06:06.076436+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:06:06.076436+0000","last_undegraded":"2024-03-09T00:06:06.076436+0000","last_fullsized":"2024-03-09T00:06:06.076436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079285+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:06:06.079285+0000","last_peered":"2024-03-09T00:06:06.079285+0000","last_clean":"2024-03-09T00:06:06.079285+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:06:06.079285+0000","last_undegraded":"2024-03-09T00:06:06.079285+0000","last_fullsized":"2024-03-09T00:06:06.079285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075921+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:06:06.075921+0000","last_peered":"2024-03-09T00:06:06.075921+0000","last_clean":"2024-03-09T00:06:06.075921+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:06:06.075921+0000","last_undegraded":"2024-03-09T00:06:06.075921+0000","last_fullsized":"2024-03-09T00:06:06.075921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079599+0000","last_change":"2024-03-09T00:05:05.059283+0000","last_active":"2024-03-09T00:06:06.079599+0000","last_peered":"2024-03-09T00:06:06.079599+0000","last_clean":"2024-03-09T00:06:06.079599+0000","last_became_active":"2024-03-09T00:05:05.058822+0000","last_became_peered":"2024-03-09T00:05:05.058822+0000","last_unstale":"2024-03-09T00:06:06.079599+0000","last_undegraded":"2024-03-09T00:06:06.079599+0000","last_fullsized":"2024-03-09T00:06:06.079599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:13:02.253810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276881+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:06:06.276881+0000","last_peered":"2024-03-09T00:06:06.276881+0000","last_clean":"2024-03-09T00:06:06.276881+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:06:06.276881+0000","last_undegraded":"2024-03-09T00:06:06.276881+0000","last_fullsized":"2024-03-09T00:06:06.276881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271661+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:06:06.271661+0000","last_peered":"2024-03-09T00:06:06.271661+0000","last_clean":"2024-03-09T00:06:06.271661+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:06:06.271661+0000","last_undegraded":"2024-03-09T00:06:06.271661+0000","last_fullsized":"2024-03-09T00:06:06.271661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079467+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:06:06.079467+0000","last_peered":"2024-03-09T00:06:06.079467+0000","last_clean":"2024-03-09T00:06:06.079467+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:06:06.079467+0000","last_undegraded":"2024-03-09T00:06:06.079467+0000","last_fullsized":"2024-03-09T00:06:06.079467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076789+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:06:06.076789+0000","last_peered":"2024-03-09T00:06:06.076789+0000","last_clean":"2024-03-09T00:06:06.076789+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:06:06.076789+0000","last_undegraded":"2024-03-09T00:06:06.076789+0000","last_fullsized":"2024-03-09T00:06:06.076789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.089907+0000","last_change":"2024-03-09T00:06:06.089420+0000","last_active":"2024-03-09T00:06:06.089907+0000","last_peered":"2024-03-09T00:06:06.089907+0000","last_clean":"2024-03-09T00:06:06.089907+0000","last_became_active":"2024-03-09T00:06:06.083164+0000","last_became_peered":"2024-03-09T00:06:06.083164+0000","last_unstale":"2024-03-09T00:06:06.089907+0000","last_undegraded":"2024-03-09T00:06:06.089907+0000","last_fullsized":"2024-03-09T00:06:06.089907+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:08:34.770550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075880+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:06:06.075880+0000","last_peered":"2024-03-09T00:06:06.075880+0000","last_clean":"2024-03-09T00:06:06.075880+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:06:06.075880+0000","last_undegraded":"2024-03-09T00:06:06.075880+0000","last_fullsized":"2024-03-09T00:06:06.075880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075786+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:06:06.075786+0000","last_peered":"2024-03-09T00:06:06.075786+0000","last_clean":"2024-03-09T00:06:06.075786+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:06:06.075786+0000","last_undegraded":"2024-03-09T00:06:06.075786+0000","last_fullsized":"2024-03-09T00:06:06.075786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082961+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:06:06.082961+0000","last_peered":"2024-03-09T00:06:06.082961+0000","last_clean":"2024-03-09T00:06:06.082961+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:06:06.082961+0000","last_undegraded":"2024-03-09T00:06:06.082961+0000","last_fullsized":"2024-03-09T00:06:06.082961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079668+0000","last_change":"2024-03-09T00:05:05.138598+0000","last_active":"2024-03-09T00:06:06.079668+0000","last_peered":"2024-03-09T00:06:06.079668+0000","last_clean":"2024-03-09T00:06:06.079668+0000","last_became_active":"2024-03-09T00:05:05.138166+0000","last_became_peered":"2024-03-09T00:05:05.138166+0000","last_unstale":"2024-03-09T00:06:06.079668+0000","last_undegraded":"2024-03-09T00:06:06.079668+0000","last_fullsized":"2024-03-09T00:06:06.079668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:59:42.864722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082189+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:06:06.082189+0000","last_peered":"2024-03-09T00:06:06.082189+0000","last_clean":"2024-03-09T00:06:06.082189+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:06:06.082189+0000","last_undegraded":"2024-03-09T00:06:06.082189+0000","last_fullsized":"2024-03-09T00:06:06.082189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276948+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:06:06.276948+0000","last_peered":"2024-03-09T00:06:06.276948+0000","last_clean":"2024-03-09T00:06:06.276948+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:06:06.276948+0000","last_undegraded":"2024-03-09T00:06:06.276948+0000","last_fullsized":"2024-03-09T00:06:06.276948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075133+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:06:06.075133+0000","last_peered":"2024-03-09T00:06:06.075133+0000","last_clean":"2024-03-09T00:06:06.075133+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:06:06.075133+0000","last_undegraded":"2024-03-09T00:06:06.075133+0000","last_fullsized":"2024-03-09T00:06:06.075133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076732+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:06:06.076732+0000","last_peered":"2024-03-09T00:06:06.076732+0000","last_clean":"2024-03-09T00:06:06.076732+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:06:06.076732+0000","last_undegraded":"2024-03-09T00:06:06.076732+0000","last_fullsized":"2024-03-09T00:06:06.076732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079746+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:06:06.079746+0000","last_peered":"2024-03-09T00:06:06.079746+0000","last_clean":"2024-03-09T00:06:06.079746+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:06:06.079746+0000","last_undegraded":"2024-03-09T00:06:06.079746+0000","last_fullsized":"2024-03-09T00:06:06.079746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075373+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:06:06.075373+0000","last_peered":"2024-03-09T00:06:06.075373+0000","last_clean":"2024-03-09T00:06:06.075373+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:06:06.075373+0000","last_undegraded":"2024-03-09T00:06:06.075373+0000","last_fullsized":"2024-03-09T00:06:06.075373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076899+0000","last_change":"2024-03-09T00:05:05.138164+0000","last_active":"2024-03-09T00:06:06.076899+0000","last_peered":"2024-03-09T00:06:06.076899+0000","last_clean":"2024-03-09T00:06:06.076899+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.076899+0000","last_undegraded":"2024-03-09T00:06:06.076899+0000","last_fullsized":"2024-03-09T00:06:06.076899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:12:44.310276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076054+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:06:06.076054+0000","last_peered":"2024-03-09T00:06:06.076054+0000","last_clean":"2024-03-09T00:06:06.076054+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.076054+0000","last_undegraded":"2024-03-09T00:06:06.076054+0000","last_fullsized":"2024-03-09T00:06:06.076054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076336+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:06:06.076336+0000","last_peered":"2024-03-09T00:06:06.076336+0000","last_clean":"2024-03-09T00:06:06.076336+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:06:06.076336+0000","last_undegraded":"2024-03-09T00:06:06.076336+0000","last_fullsized":"2024-03-09T00:06:06.076336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079810+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:06:06.079810+0000","last_peered":"2024-03-09T00:06:06.079810+0000","last_clean":"2024-03-09T00:06:06.079810+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:06:06.079810+0000","last_undegraded":"2024-03-09T00:06:06.079810+0000","last_fullsized":"2024-03-09T00:06:06.079810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276945+0000","last_change":"2024-03-09T00:05:05.058253+0000","last_active":"2024-03-09T00:06:06.276945+0000","last_peered":"2024-03-09T00:06:06.276945+0000","last_clean":"2024-03-09T00:06:06.276945+0000","last_became_active":"2024-03-09T00:05:05.057864+0000","last_became_peered":"2024-03-09T00:05:05.057864+0000","last_unstale":"2024-03-09T00:06:06.276945+0000","last_undegraded":"2024-03-09T00:06:06.276945+0000","last_fullsized":"2024-03-09T00:06:06.276945+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:33:14.993355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271946+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:06:06.271946+0000","last_peered":"2024-03-09T00:06:06.271946+0000","last_clean":"2024-03-09T00:06:06.271946+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:06:06.271946+0000","last_undegraded":"2024-03-09T00:06:06.271946+0000","last_fullsized":"2024-03-09T00:06:06.271946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079321+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:06:06.079321+0000","last_peered":"2024-03-09T00:06:06.079321+0000","last_clean":"2024-03-09T00:06:06.079321+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:06:06.079321+0000","last_undegraded":"2024-03-09T00:06:06.079321+0000","last_fullsized":"2024-03-09T00:06:06.079321+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076429+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:06:06.076429+0000","last_peered":"2024-03-09T00:06:06.076429+0000","last_clean":"2024-03-09T00:06:06.076429+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:06:06.076429+0000","last_undegraded":"2024-03-09T00:06:06.076429+0000","last_fullsized":"2024-03-09T00:06:06.076429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076267+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:06:06.076267+0000","last_peered":"2024-03-09T00:06:06.076267+0000","last_clean":"2024-03-09T00:06:06.076267+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:06:06.076267+0000","last_undegraded":"2024-03-09T00:06:06.076267+0000","last_fullsized":"2024-03-09T00:06:06.076267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079600+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:06:06.079600+0000","last_peered":"2024-03-09T00:06:06.079600+0000","last_clean":"2024-03-09T00:06:06.079600+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:06:06.079600+0000","last_undegraded":"2024-03-09T00:06:06.079600+0000","last_fullsized":"2024-03-09T00:06:06.079600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079335+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:06:06.079335+0000","last_peered":"2024-03-09T00:06:06.079335+0000","last_clean":"2024-03-09T00:06:06.079335+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:06:06.079335+0000","last_undegraded":"2024-03-09T00:06:06.079335+0000","last_fullsized":"2024-03-09T00:06:06.079335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.278451+0000","last_change":"2024-03-09T00:06:06.278068+0000","last_active":"2024-03-09T00:06:06.278451+0000","last_peered":"2024-03-09T00:06:06.278451+0000","last_clean":"2024-03-09T00:06:06.278451+0000","last_became_active":"2024-03-09T00:06:06.277772+0000","last_became_peered":"2024-03-09T00:06:06.277772+0000","last_unstale":"2024-03-09T00:06:06.278451+0000","last_undegraded":"2024-03-09T00:06:06.278451+0000","last_fullsized":"2024-03-09T00:06:06.278451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:36:33.637238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076841+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:06:06.076841+0000","last_peered":"2024-03-09T00:06:06.076841+0000","last_clean":"2024-03-09T00:06:06.076841+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:06:06.076841+0000","last_undegraded":"2024-03-09T00:06:06.076841+0000","last_fullsized":"2024-03-09T00:06:06.076841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921495+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:06:06.921495+0000","last_peered":"2024-03-09T00:06:06.921495+0000","last_clean":"2024-03-09T00:06:06.921495+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:06:06.921495+0000","last_undegraded":"2024-03-09T00:06:06.921495+0000","last_fullsized":"2024-03-09T00:06:06.921495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'8","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:23.979195+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:06:23.979195+0000","last_peered":"2024-03-09T00:06:23.979195+0000","last_clean":"2024-03-09T00:06:23.979195+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:06:23.979195+0000","last_undegraded":"2024-03-09T00:06:23.979195+0000","last_fullsized":"2024-03-09T00:06:23.979195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921938+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:06:06.921938+0000","last_peered":"2024-03-09T00:06:06.921938+0000","last_clean":"2024-03-09T00:06:06.921938+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:06:06.921938+0000","last_undegraded":"2024-03-09T00:06:06.921938+0000","last_fullsized":"2024-03-09T00:06:06.921938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081216+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:06:06.081216+0000","last_peered":"2024-03-09T00:06:06.081216+0000","last_clean":"2024-03-09T00:06:06.081216+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:06:06.081216+0000","last_undegraded":"2024-03-09T00:06:06.081216+0000","last_fullsized":"2024-03-09T00:06:06.081216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272193+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:06:06.272193+0000","last_peered":"2024-03-09T00:06:06.272193+0000","last_clean":"2024-03-09T00:06:06.272193+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:06:06.272193+0000","last_undegraded":"2024-03-09T00:06:06.272193+0000","last_fullsized":"2024-03-09T00:06:06.272193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081386+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:06:06.081386+0000","last_peered":"2024-03-09T00:06:06.081386+0000","last_clean":"2024-03-09T00:06:06.081386+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:06:06.081386+0000","last_undegraded":"2024-03-09T00:06:06.081386+0000","last_fullsized":"2024-03-09T00:06:06.081386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271932+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:06:06.271932+0000","last_peered":"2024-03-09T00:06:06.271932+0000","last_clean":"2024-03-09T00:06:06.271932+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:06:06.271932+0000","last_undegraded":"2024-03-09T00:06:06.271932+0000","last_fullsized":"2024-03-09T00:06:06.271932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513561+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:06:06.513561+0000","last_peered":"2024-03-09T00:06:06.513561+0000","last_clean":"2024-03-09T00:06:06.513561+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:06:06.513561+0000","last_undegraded":"2024-03-09T00:06:06.513561+0000","last_fullsized":"2024-03-09T00:06:06.513561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081820+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:06:06.081820+0000","last_peered":"2024-03-09T00:06:06.081820+0000","last_clean":"2024-03-09T00:06:06.081820+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:06:06.081820+0000","last_undegraded":"2024-03-09T00:06:06.081820+0000","last_fullsized":"2024-03-09T00:06:06.081820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277174+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:06:06.277174+0000","last_peered":"2024-03-09T00:06:06.277174+0000","last_clean":"2024-03-09T00:06:06.277174+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:06:06.277174+0000","last_undegraded":"2024-03-09T00:06:06.277174+0000","last_fullsized":"2024-03-09T00:06:06.277174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921579+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:06:06.921579+0000","last_peered":"2024-03-09T00:06:06.921579+0000","last_clean":"2024-03-09T00:06:06.921579+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:06:06.921579+0000","last_undegraded":"2024-03-09T00:06:06.921579+0000","last_fullsized":"2024-03-09T00:06:06.921579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079350+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:06:06.079350+0000","last_peered":"2024-03-09T00:06:06.079350+0000","last_clean":"2024-03-09T00:06:06.079350+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:06:06.079350+0000","last_undegraded":"2024-03-09T00:06:06.079350+0000","last_fullsized":"2024-03-09T00:06:06.079350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271971+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:06:06.271971+0000","last_peered":"2024-03-09T00:06:06.271971+0000","last_clean":"2024-03-09T00:06:06.271971+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:06:06.271971+0000","last_undegraded":"2024-03-09T00:06:06.271971+0000","last_fullsized":"2024-03-09T00:06:06.271971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081894+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:06:06.081894+0000","last_peered":"2024-03-09T00:06:06.081894+0000","last_clean":"2024-03-09T00:06:06.081894+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:06:06.081894+0000","last_undegraded":"2024-03-09T00:06:06.081894+0000","last_fullsized":"2024-03-09T00:06:06.081894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921869+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:06:06.921869+0000","last_peered":"2024-03-09T00:06:06.921869+0000","last_clean":"2024-03-09T00:06:06.921869+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:06:06.921869+0000","last_undegraded":"2024-03-09T00:06:06.921869+0000","last_fullsized":"2024-03-09T00:06:06.921869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271986+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:06:06.271986+0000","last_peered":"2024-03-09T00:06:06.271986+0000","last_clean":"2024-03-09T00:06:06.271986+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:06:06.271986+0000","last_undegraded":"2024-03-09T00:06:06.271986+0000","last_fullsized":"2024-03-09T00:06:06.271986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082328+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:06:06.082328+0000","last_peered":"2024-03-09T00:06:06.082328+0000","last_clean":"2024-03-09T00:06:06.082328+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:06:06.082328+0000","last_undegraded":"2024-03-09T00:06:06.082328+0000","last_fullsized":"2024-03-09T00:06:06.082328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079588+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:06:06.079588+0000","last_peered":"2024-03-09T00:06:06.079588+0000","last_clean":"2024-03-09T00:06:06.079588+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:06:06.079588+0000","last_undegraded":"2024-03-09T00:06:06.079588+0000","last_fullsized":"2024-03-09T00:06:06.079588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513345+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:06:06.513345+0000","last_peered":"2024-03-09T00:06:06.513345+0000","last_clean":"2024-03-09T00:06:06.513345+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:06:06.513345+0000","last_undegraded":"2024-03-09T00:06:06.513345+0000","last_fullsized":"2024-03-09T00:06:06.513345+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081834+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:06:06.081834+0000","last_peered":"2024-03-09T00:06:06.081834+0000","last_clean":"2024-03-09T00:06:06.081834+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:06:06.081834+0000","last_undegraded":"2024-03-09T00:06:06.081834+0000","last_fullsized":"2024-03-09T00:06:06.081834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079996+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:06:06.079996+0000","last_peered":"2024-03-09T00:06:06.079996+0000","last_clean":"2024-03-09T00:06:06.079996+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:06:06.079996+0000","last_undegraded":"2024-03-09T00:06:06.079996+0000","last_fullsized":"2024-03-09T00:06:06.079996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081211+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:06:06.081211+0000","last_peered":"2024-03-09T00:06:06.081211+0000","last_clean":"2024-03-09T00:06:06.081211+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:06:06.081211+0000","last_undegraded":"2024-03-09T00:06:06.081211+0000","last_fullsized":"2024-03-09T00:06:06.081211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076463+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:06:06.076463+0000","last_peered":"2024-03-09T00:06:06.076463+0000","last_clean":"2024-03-09T00:06:06.076463+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:06:06.076463+0000","last_undegraded":"2024-03-09T00:06:06.076463+0000","last_fullsized":"2024-03-09T00:06:06.076463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079730+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:06:06.079730+0000","last_peered":"2024-03-09T00:06:06.079730+0000","last_clean":"2024-03-09T00:06:06.079730+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:06:06.079730+0000","last_undegraded":"2024-03-09T00:06:06.079730+0000","last_fullsized":"2024-03-09T00:06:06.079730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075532+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:06:06.075532+0000","last_peered":"2024-03-09T00:06:06.075532+0000","last_clean":"2024-03-09T00:06:06.075532+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:06:06.075532+0000","last_undegraded":"2024-03-09T00:06:06.075532+0000","last_fullsized":"2024-03-09T00:06:06.075532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076859+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:06:06.076859+0000","last_peered":"2024-03-09T00:06:06.076859+0000","last_clean":"2024-03-09T00:06:06.076859+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:06:06.076859+0000","last_undegraded":"2024-03-09T00:06:06.076859+0000","last_fullsized":"2024-03-09T00:06:06.076859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080006+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:06:06.080006+0000","last_peered":"2024-03-09T00:06:06.080006+0000","last_clean":"2024-03-09T00:06:06.080006+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:06:06.080006+0000","last_undegraded":"2024-03-09T00:06:06.080006+0000","last_fullsized":"2024-03-09T00:06:06.080006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079554+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:06:06.079554+0000","last_peered":"2024-03-09T00:06:06.079554+0000","last_clean":"2024-03-09T00:06:06.079554+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:06:06.079554+0000","last_undegraded":"2024-03-09T00:06:06.079554+0000","last_fullsized":"2024-03-09T00:06:06.079554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081561+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:06:06.081561+0000","last_peered":"2024-03-09T00:06:06.081561+0000","last_clean":"2024-03-09T00:06:06.081561+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:06:06.081561+0000","last_undegraded":"2024-03-09T00:06:06.081561+0000","last_fullsized":"2024-03-09T00:06:06.081561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081789+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:06:06.081789+0000","last_peered":"2024-03-09T00:06:06.081789+0000","last_clean":"2024-03-09T00:06:06.081789+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:06:06.081789+0000","last_undegraded":"2024-03-09T00:06:06.081789+0000","last_fullsized":"2024-03-09T00:06:06.081789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081489+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:06:06.081489+0000","last_peered":"2024-03-09T00:06:06.081489+0000","last_clean":"2024-03-09T00:06:06.081489+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:06:06.081489+0000","last_undegraded":"2024-03-09T00:06:06.081489+0000","last_fullsized":"2024-03-09T00:06:06.081489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076818+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:06:06.076818+0000","last_peered":"2024-03-09T00:06:06.076818+0000","last_clean":"2024-03-09T00:06:06.076818+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:06:06.076818+0000","last_undegraded":"2024-03-09T00:06:06.076818+0000","last_fullsized":"2024-03-09T00:06:06.076818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079815+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:06:06.079815+0000","last_peered":"2024-03-09T00:06:06.079815+0000","last_clean":"2024-03-09T00:06:06.079815+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:06:06.079815+0000","last_undegraded":"2024-03-09T00:06:06.079815+0000","last_fullsized":"2024-03-09T00:06:06.079815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082078+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:06:06.082078+0000","last_peered":"2024-03-09T00:06:06.082078+0000","last_clean":"2024-03-09T00:06:06.082078+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:06:06.082078+0000","last_undegraded":"2024-03-09T00:06:06.082078+0000","last_fullsized":"2024-03-09T00:06:06.082078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921924+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:06:06.921924+0000","last_peered":"2024-03-09T00:06:06.921924+0000","last_clean":"2024-03-09T00:06:06.921924+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:06:06.921924+0000","last_undegraded":"2024-03-09T00:06:06.921924+0000","last_fullsized":"2024-03-09T00:06:06.921924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076018+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:06:06.076018+0000","last_peered":"2024-03-09T00:06:06.076018+0000","last_clean":"2024-03-09T00:06:06.076018+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:06:06.076018+0000","last_undegraded":"2024-03-09T00:06:06.076018+0000","last_fullsized":"2024-03-09T00:06:06.076018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921769+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:06:06.921769+0000","last_peered":"2024-03-09T00:06:06.921769+0000","last_clean":"2024-03-09T00:06:06.921769+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:06:06.921769+0000","last_undegraded":"2024-03-09T00:06:06.921769+0000","last_fullsized":"2024-03-09T00:06:06.921769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513579+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:06:06.513579+0000","last_peered":"2024-03-09T00:06:06.513579+0000","last_clean":"2024-03-09T00:06:06.513579+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:06:06.513579+0000","last_undegraded":"2024-03-09T00:06:06.513579+0000","last_fullsized":"2024-03-09T00:06:06.513579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076412+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:06:06.076412+0000","last_peered":"2024-03-09T00:06:06.076412+0000","last_clean":"2024-03-09T00:06:06.076412+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:06:06.076412+0000","last_undegraded":"2024-03-09T00:06:06.076412+0000","last_fullsized":"2024-03-09T00:06:06.076412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079497+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:06:06.079497+0000","last_peered":"2024-03-09T00:06:06.079497+0000","last_clean":"2024-03-09T00:06:06.079497+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:06:06.079497+0000","last_undegraded":"2024-03-09T00:06:06.079497+0000","last_fullsized":"2024-03-09T00:06:06.079497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70084,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":258,"num_read_kb":200,"num_write":89,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":199198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":519110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32868,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712284,"statfs":{"total":95995035648,"available":95961378816,"internally_reserved":0,"allocated":1732608,"data_stored":1174862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":523788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":519714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1114156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":515014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":578927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":515104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":578653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":515118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":519132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1658880,"data_stored":1109660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:06:40.460 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T00:06:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:40 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/439267907' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:40 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3798805855' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:40 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/929869091' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 -- 172.21.15.73:0/3089704689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260101e10 msgr2=0x7ff260104200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 --2- 172.21.15.73:0/3089704689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260101e10 0x7ff260104200 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7ff250009900 tx=0x7ff25002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 -- 172.21.15.73:0/3089704689 shutdown_connections 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 --2- 172.21.15.73:0/3089704689 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff260107070 0x7ff2601094c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 --2- 172.21.15.73:0/3089704689 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff260104740 0x7ff260106b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 --2- 172.21.15.73:0/3089704689 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260101e10 0x7ff260104200 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 -- 172.21.15.73:0/3089704689 >> 172.21.15.73:0/3089704689 conn(0x7ff2600fb820 msgr2=0x7ff2600fdc40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 -- 172.21.15.73:0/3089704689 shutdown_connections 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 -- 172.21.15.73:0/3089704689 wait complete. 2024-03-09T00:06:40.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.602+0000 7ff265797640 1 Processor -- start 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 -- start start 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff260101e10 0x7ff260197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff260107070 0x7ff26019ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26010d000 con 0x7ff260104740 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26010ce80 con 0x7ff260101e10 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff265797640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff26010d180 con 0x7ff260107070 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60202/0 (socket says 172.21.15.73:60202) 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 -- 172.21.15.73:0/1567905045 learned_addr learned my addr 172.21.15.73:0/1567905045 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:40.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 -- 172.21.15.73:0/1567905045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff260107070 msgr2=0x7ff26019ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff260107070 0x7ff26019ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 -- 172.21.15.73:0/1567905045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff260101e10 msgr2=0x7ff260197620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff260101e10 0x7ff260197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 -- 172.21.15.73:0/1567905045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff26019f240 con 0x7ff260104740 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff25e7fc640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c970 tx=0x7ff25400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.603+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff254007bb0 con 0x7ff260104740 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.604+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff26019f530 con 0x7ff260104740 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.604+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff254007d50 con 0x7ff260104740 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.604+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff25400f450 con 0x7ff260104740 2024-03-09T00:06:40.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.604+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff26019fad0 con 0x7ff260104740 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.604+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2600fd310 con 0x7ff260104740 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.605+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff254004f90 con 0x7ff260104740 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.606+0000 7ff247fff640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 0x7ff2480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.606+0000 7ff25effd640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 0x7ff2480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.606+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff2540cec20 con 0x7ff260104740 2024-03-09T00:06:40.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.606+0000 7ff25effd640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 0x7ff2480a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2500057c0 tx=0x7ff250000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:40.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.610+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff254014030 con 0x7ff260104740 2024-03-09T00:06:40.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.781+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff260061730 con 0x7ff2480a2800 2024-03-09T00:06:40.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.790+0000 7ff247fff640 1 -- 172.21.15.73:0/1567905045 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340392 (secure 0 0 0) 0x7ff260061730 con 0x7ff2480a2800 2024-03-09T00:06:40.891 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 msgr2=0x7ff2480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 0x7ff2480a4cc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7ff2500057c0 tx=0x7ff250000bc0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 msgr2=0x7ff260197b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7ff25400c970 tx=0x7ff25400ce30 comp rx=0 tx=0).stop 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 shutdown_connections 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff2480a2800 0x7ff2480a4cc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff260107070 0x7ff26019ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff260104740 0x7ff260197b60 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 --2- 172.21.15.73:0/1567905045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff260101e10 0x7ff260197620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:40.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.793+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 >> 172.21.15.73:0/1567905045 conn(0x7ff2600fb820 msgr2=0x7ff260102770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:40.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.794+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 shutdown_connections 2024-03-09T00:06:40.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:40.794+0000 7ff265797640 1 -- 172.21.15.73:0/1567905045 wait complete. 2024-03-09T00:06:40.895 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-09T00:06:39.967346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660686,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":394,"num_read_kb":319,"num_write":272,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":715,"ondisk_log_size":715,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396832,"kb_used_data":14340,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544992,"statfs":{"total":1151940427776,"available":1151534071808,"internally_reserved":0,"allocated":14684160,"data_stored":8183238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079857+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:06:06.079857+0000","last_peered":"2024-03-09T00:06:06.079857+0000","last_clean":"2024-03-09T00:06:06.079857+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:06:06.079857+0000","last_undegraded":"2024-03-09T00:06:06.079857+0000","last_fullsized":"2024-03-09T00:06:06.079857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081264+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:06:06.081264+0000","last_peered":"2024-03-09T00:06:06.081264+0000","last_clean":"2024-03-09T00:06:06.081264+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:06:06.081264+0000","last_undegraded":"2024-03-09T00:06:06.081264+0000","last_fullsized":"2024-03-09T00:06:06.081264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081167+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:06:06.081167+0000","last_peered":"2024-03-09T00:06:06.081167+0000","last_clean":"2024-03-09T00:06:06.081167+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:06:06.081167+0000","last_undegraded":"2024-03-09T00:06:06.081167+0000","last_fullsized":"2024-03-09T00:06:06.081167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272147+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:06:06.272147+0000","last_peered":"2024-03-09T00:06:06.272147+0000","last_clean":"2024-03-09T00:06:06.272147+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:06:06.272147+0000","last_undegraded":"2024-03-09T00:06:06.272147+0000","last_fullsized":"2024-03-09T00:06:06.272147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081462+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:06:06.081462+0000","last_peered":"2024-03-09T00:06:06.081462+0000","last_clean":"2024-03-09T00:06:06.081462+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:06:06.081462+0000","last_undegraded":"2024-03-09T00:06:06.081462+0000","last_fullsized":"2024-03-09T00:06:06.081462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075196+0000","last_change":"2024-03-09T00:05:05.058033+0000","last_active":"2024-03-09T00:06:06.075196+0000","last_peered":"2024-03-09T00:06:06.075196+0000","last_clean":"2024-03-09T00:06:06.075196+0000","last_became_active":"2024-03-09T00:05:05.057575+0000","last_became_peered":"2024-03-09T00:05:05.057575+0000","last_unstale":"2024-03-09T00:06:06.075196+0000","last_undegraded":"2024-03-09T00:06:06.075196+0000","last_fullsized":"2024-03-09T00:06:06.075196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:35:58.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079431+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:06:06.079431+0000","last_peered":"2024-03-09T00:06:06.079431+0000","last_clean":"2024-03-09T00:06:06.079431+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:06:06.079431+0000","last_undegraded":"2024-03-09T00:06:06.079431+0000","last_fullsized":"2024-03-09T00:06:06.079431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921426+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:06:06.921426+0000","last_peered":"2024-03-09T00:06:06.921426+0000","last_clean":"2024-03-09T00:06:06.921426+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:06:06.921426+0000","last_undegraded":"2024-03-09T00:06:06.921426+0000","last_fullsized":"2024-03-09T00:06:06.921426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513955+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:06:06.513955+0000","last_peered":"2024-03-09T00:06:06.513955+0000","last_clean":"2024-03-09T00:06:06.513955+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:06:06.513955+0000","last_undegraded":"2024-03-09T00:06:06.513955+0000","last_fullsized":"2024-03-09T00:06:06.513955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075525+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:06:06.075525+0000","last_peered":"2024-03-09T00:06:06.075525+0000","last_clean":"2024-03-09T00:06:06.075525+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:06:06.075525+0000","last_undegraded":"2024-03-09T00:06:06.075525+0000","last_fullsized":"2024-03-09T00:06:06.075525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079896+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:06:06.079896+0000","last_peered":"2024-03-09T00:06:06.079896+0000","last_clean":"2024-03-09T00:06:06.079896+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:06:06.079896+0000","last_undegraded":"2024-03-09T00:06:06.079896+0000","last_fullsized":"2024-03-09T00:06:06.079896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271903+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:06:06.271903+0000","last_peered":"2024-03-09T00:06:06.271903+0000","last_clean":"2024-03-09T00:06:06.271903+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:06:06.271903+0000","last_undegraded":"2024-03-09T00:06:06.271903+0000","last_fullsized":"2024-03-09T00:06:06.271903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076426+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:06:06.076426+0000","last_peered":"2024-03-09T00:06:06.076426+0000","last_clean":"2024-03-09T00:06:06.076426+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:06:06.076426+0000","last_undegraded":"2024-03-09T00:06:06.076426+0000","last_fullsized":"2024-03-09T00:06:06.076426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081033+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:06:06.081033+0000","last_peered":"2024-03-09T00:06:06.081033+0000","last_clean":"2024-03-09T00:06:06.081033+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:06:06.081033+0000","last_undegraded":"2024-03-09T00:06:06.081033+0000","last_fullsized":"2024-03-09T00:06:06.081033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080896+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:06:06.080896+0000","last_peered":"2024-03-09T00:06:06.080896+0000","last_clean":"2024-03-09T00:06:06.080896+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:06:06.080896+0000","last_undegraded":"2024-03-09T00:06:06.080896+0000","last_fullsized":"2024-03-09T00:06:06.080896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513424+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:06:06.513424+0000","last_peered":"2024-03-09T00:06:06.513424+0000","last_clean":"2024-03-09T00:06:06.513424+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:06:06.513424+0000","last_undegraded":"2024-03-09T00:06:06.513424+0000","last_fullsized":"2024-03-09T00:06:06.513424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079620+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:06:06.079620+0000","last_peered":"2024-03-09T00:06:06.079620+0000","last_clean":"2024-03-09T00:06:06.079620+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:06:06.079620+0000","last_undegraded":"2024-03-09T00:06:06.079620+0000","last_fullsized":"2024-03-09T00:06:06.079620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277255+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:06:06.277255+0000","last_peered":"2024-03-09T00:06:06.277255+0000","last_clean":"2024-03-09T00:06:06.277255+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:06:06.277255+0000","last_undegraded":"2024-03-09T00:06:06.277255+0000","last_fullsized":"2024-03-09T00:06:06.277255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081723+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:06:06.081723+0000","last_peered":"2024-03-09T00:06:06.081723+0000","last_clean":"2024-03-09T00:06:06.081723+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:06:06.081723+0000","last_undegraded":"2024-03-09T00:06:06.081723+0000","last_fullsized":"2024-03-09T00:06:06.081723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081285+0000","last_change":"2024-03-09T00:05:05.138220+0000","last_active":"2024-03-09T00:06:06.081285+0000","last_peered":"2024-03-09T00:06:06.081285+0000","last_clean":"2024-03-09T00:06:06.081285+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.081285+0000","last_undegraded":"2024-03-09T00:06:06.081285+0000","last_fullsized":"2024-03-09T00:06:06.081285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:41.455601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277194+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:06:06.277194+0000","last_peered":"2024-03-09T00:06:06.277194+0000","last_clean":"2024-03-09T00:06:06.277194+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:06:06.277194+0000","last_undegraded":"2024-03-09T00:06:06.277194+0000","last_fullsized":"2024-03-09T00:06:06.277194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922002+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:06:06.922002+0000","last_peered":"2024-03-09T00:06:06.922002+0000","last_clean":"2024-03-09T00:06:06.922002+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.922002+0000","last_undegraded":"2024-03-09T00:06:06.922002+0000","last_fullsized":"2024-03-09T00:06:06.922002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271466+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:06:06.271466+0000","last_peered":"2024-03-09T00:06:06.271466+0000","last_clean":"2024-03-09T00:06:06.271466+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:06:06.271466+0000","last_undegraded":"2024-03-09T00:06:06.271466+0000","last_fullsized":"2024-03-09T00:06:06.271466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922049+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:06:06.922049+0000","last_peered":"2024-03-09T00:06:06.922049+0000","last_clean":"2024-03-09T00:06:06.922049+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:06:06.922049+0000","last_undegraded":"2024-03-09T00:06:06.922049+0000","last_fullsized":"2024-03-09T00:06:06.922049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081306+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:06:06.081306+0000","last_peered":"2024-03-09T00:06:06.081306+0000","last_clean":"2024-03-09T00:06:06.081306+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:06:06.081306+0000","last_undegraded":"2024-03-09T00:06:06.081306+0000","last_fullsized":"2024-03-09T00:06:06.081306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271674+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:06:06.271674+0000","last_peered":"2024-03-09T00:06:06.271674+0000","last_clean":"2024-03-09T00:06:06.271674+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:06:06.271674+0000","last_undegraded":"2024-03-09T00:06:06.271674+0000","last_fullsized":"2024-03-09T00:06:06.271674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082791+0000","last_change":"2024-03-09T00:05:05.142560+0000","last_active":"2024-03-09T00:06:06.082791+0000","last_peered":"2024-03-09T00:06:06.082791+0000","last_clean":"2024-03-09T00:06:06.082791+0000","last_became_active":"2024-03-09T00:05:05.139877+0000","last_became_peered":"2024-03-09T00:05:05.139877+0000","last_unstale":"2024-03-09T00:06:06.082791+0000","last_undegraded":"2024-03-09T00:06:06.082791+0000","last_fullsized":"2024-03-09T00:06:06.082791+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:30.441723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513607+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:06:06.513607+0000","last_peered":"2024-03-09T00:06:06.513607+0000","last_clean":"2024-03-09T00:06:06.513607+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:06:06.513607+0000","last_undegraded":"2024-03-09T00:06:06.513607+0000","last_fullsized":"2024-03-09T00:06:06.513607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271559+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:06:06.271559+0000","last_peered":"2024-03-09T00:06:06.271559+0000","last_clean":"2024-03-09T00:06:06.271559+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:06:06.271559+0000","last_undegraded":"2024-03-09T00:06:06.271559+0000","last_fullsized":"2024-03-09T00:06:06.271559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082413+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:06:06.082413+0000","last_peered":"2024-03-09T00:06:06.082413+0000","last_clean":"2024-03-09T00:06:06.082413+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:06:06.082413+0000","last_undegraded":"2024-03-09T00:06:06.082413+0000","last_fullsized":"2024-03-09T00:06:06.082413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076341+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:06:06.076341+0000","last_peered":"2024-03-09T00:06:06.076341+0000","last_clean":"2024-03-09T00:06:06.076341+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:06:06.076341+0000","last_undegraded":"2024-03-09T00:06:06.076341+0000","last_fullsized":"2024-03-09T00:06:06.076341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081523+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:06:06.081523+0000","last_peered":"2024-03-09T00:06:06.081523+0000","last_clean":"2024-03-09T00:06:06.081523+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:06:06.081523+0000","last_undegraded":"2024-03-09T00:06:06.081523+0000","last_fullsized":"2024-03-09T00:06:06.081523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075947+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:06:06.075947+0000","last_peered":"2024-03-09T00:06:06.075947+0000","last_clean":"2024-03-09T00:06:06.075947+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:06:06.075947+0000","last_undegraded":"2024-03-09T00:06:06.075947+0000","last_fullsized":"2024-03-09T00:06:06.075947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082037+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:06:06.082037+0000","last_peered":"2024-03-09T00:06:06.082037+0000","last_clean":"2024-03-09T00:06:06.082037+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:06:06.082037+0000","last_undegraded":"2024-03-09T00:06:06.082037+0000","last_fullsized":"2024-03-09T00:06:06.082037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081531+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:06:06.081531+0000","last_peered":"2024-03-09T00:06:06.081531+0000","last_clean":"2024-03-09T00:06:06.081531+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:06:06.081531+0000","last_undegraded":"2024-03-09T00:06:06.081531+0000","last_fullsized":"2024-03-09T00:06:06.081531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075185+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:06:06.075185+0000","last_peered":"2024-03-09T00:06:06.075185+0000","last_clean":"2024-03-09T00:06:06.075185+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:06:06.075185+0000","last_undegraded":"2024-03-09T00:06:06.075185+0000","last_fullsized":"2024-03-09T00:06:06.075185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076123+0000","last_change":"2024-03-09T00:05:05.056415+0000","last_active":"2024-03-09T00:06:06.076123+0000","last_peered":"2024-03-09T00:06:06.076123+0000","last_clean":"2024-03-09T00:06:06.076123+0000","last_became_active":"2024-03-09T00:05:05.056025+0000","last_became_peered":"2024-03-09T00:05:05.056025+0000","last_unstale":"2024-03-09T00:06:06.076123+0000","last_undegraded":"2024-03-09T00:06:06.076123+0000","last_fullsized":"2024-03-09T00:06:06.076123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:29.970623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079682+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:06:06.079682+0000","last_peered":"2024-03-09T00:06:06.079682+0000","last_clean":"2024-03-09T00:06:06.079682+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:06:06.079682+0000","last_undegraded":"2024-03-09T00:06:06.079682+0000","last_fullsized":"2024-03-09T00:06:06.079682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076321+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:06:06.076321+0000","last_peered":"2024-03-09T00:06:06.076321+0000","last_clean":"2024-03-09T00:06:06.076321+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:06:06.076321+0000","last_undegraded":"2024-03-09T00:06:06.076321+0000","last_fullsized":"2024-03-09T00:06:06.076321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921566+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:06:06.921566+0000","last_peered":"2024-03-09T00:06:06.921566+0000","last_clean":"2024-03-09T00:06:06.921566+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:06:06.921566+0000","last_undegraded":"2024-03-09T00:06:06.921566+0000","last_fullsized":"2024-03-09T00:06:06.921566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081208+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:06:06.081208+0000","last_peered":"2024-03-09T00:06:06.081208+0000","last_clean":"2024-03-09T00:06:06.081208+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:06:06.081208+0000","last_undegraded":"2024-03-09T00:06:06.081208+0000","last_fullsized":"2024-03-09T00:06:06.081208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075906+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:06:06.075906+0000","last_peered":"2024-03-09T00:06:06.075906+0000","last_clean":"2024-03-09T00:06:06.075906+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:06:06.075906+0000","last_undegraded":"2024-03-09T00:06:06.075906+0000","last_fullsized":"2024-03-09T00:06:06.075906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276794+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:06:06.276794+0000","last_peered":"2024-03-09T00:06:06.276794+0000","last_clean":"2024-03-09T00:06:06.276794+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:06:06.276794+0000","last_undegraded":"2024-03-09T00:06:06.276794+0000","last_fullsized":"2024-03-09T00:06:06.276794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921459+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:06:06.921459+0000","last_peered":"2024-03-09T00:06:06.921459+0000","last_clean":"2024-03-09T00:06:06.921459+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:06:06.921459+0000","last_undegraded":"2024-03-09T00:06:06.921459+0000","last_fullsized":"2024-03-09T00:06:06.921459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075432+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:06:06.075432+0000","last_peered":"2024-03-09T00:06:06.075432+0000","last_clean":"2024-03-09T00:06:06.075432+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:06:06.075432+0000","last_undegraded":"2024-03-09T00:06:06.075432+0000","last_fullsized":"2024-03-09T00:06:06.075432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513829+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:06:06.513829+0000","last_peered":"2024-03-09T00:06:06.513829+0000","last_clean":"2024-03-09T00:06:06.513829+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:06:06.513829+0000","last_undegraded":"2024-03-09T00:06:06.513829+0000","last_fullsized":"2024-03-09T00:06:06.513829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081694+0000","last_change":"2024-03-09T00:05:05.139426+0000","last_active":"2024-03-09T00:06:06.081694+0000","last_peered":"2024-03-09T00:06:06.081694+0000","last_clean":"2024-03-09T00:06:06.081694+0000","last_became_active":"2024-03-09T00:05:05.138971+0000","last_became_peered":"2024-03-09T00:05:05.138971+0000","last_unstale":"2024-03-09T00:06:06.081694+0000","last_undegraded":"2024-03-09T00:06:06.081694+0000","last_fullsized":"2024-03-09T00:06:06.081694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:40:59.870972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.514952+0000","last_change":"2024-03-09T00:06:06.514952+0000","last_active":"2024-03-09T00:06:06.514952+0000","last_peered":"2024-03-09T00:06:06.514952+0000","last_clean":"2024-03-09T00:06:06.514952+0000","last_became_active":"2024-03-09T00:06:06.514480+0000","last_became_peered":"2024-03-09T00:06:06.514480+0000","last_unstale":"2024-03-09T00:06:06.514952+0000","last_undegraded":"2024-03-09T00:06:06.514952+0000","last_fullsized":"2024-03-09T00:06:06.514952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:03.976364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081301+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:06:06.081301+0000","last_peered":"2024-03-09T00:06:06.081301+0000","last_clean":"2024-03-09T00:06:06.081301+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:06:06.081301+0000","last_undegraded":"2024-03-09T00:06:06.081301+0000","last_fullsized":"2024-03-09T00:06:06.081301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271940+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:06:06.271940+0000","last_peered":"2024-03-09T00:06:06.271940+0000","last_clean":"2024-03-09T00:06:06.271940+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:06:06.271940+0000","last_undegraded":"2024-03-09T00:06:06.271940+0000","last_fullsized":"2024-03-09T00:06:06.271940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076324+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:06:06.076324+0000","last_peered":"2024-03-09T00:06:06.076324+0000","last_clean":"2024-03-09T00:06:06.076324+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:06:06.076324+0000","last_undegraded":"2024-03-09T00:06:06.076324+0000","last_fullsized":"2024-03-09T00:06:06.076324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513828+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:06:06.513828+0000","last_peered":"2024-03-09T00:06:06.513828+0000","last_clean":"2024-03-09T00:06:06.513828+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:06:06.513828+0000","last_undegraded":"2024-03-09T00:06:06.513828+0000","last_fullsized":"2024-03-09T00:06:06.513828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.074900+0000","last_change":"2024-03-09T00:05:05.058517+0000","last_active":"2024-03-09T00:06:06.074900+0000","last_peered":"2024-03-09T00:06:06.074900+0000","last_clean":"2024-03-09T00:06:06.074900+0000","last_became_active":"2024-03-09T00:05:05.057877+0000","last_became_peered":"2024-03-09T00:05:05.057877+0000","last_unstale":"2024-03-09T00:06:06.074900+0000","last_undegraded":"2024-03-09T00:06:06.074900+0000","last_fullsized":"2024-03-09T00:06:06.074900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:42:49.424790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'36","reported_seq":320,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:39.198178+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:06:39.198178+0000","last_peered":"2024-03-09T00:06:39.198178+0000","last_clean":"2024-03-09T00:06:39.198178+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:06:39.198178+0000","last_undegraded":"2024-03-09T00:06:39.198178+0000","last_fullsized":"2024-03-09T00:06:39.198178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":36,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082439+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:06:06.082439+0000","last_peered":"2024-03-09T00:06:06.082439+0000","last_clean":"2024-03-09T00:06:06.082439+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:06:06.082439+0000","last_undegraded":"2024-03-09T00:06:06.082439+0000","last_fullsized":"2024-03-09T00:06:06.082439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076594+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:06:06.076594+0000","last_peered":"2024-03-09T00:06:06.076594+0000","last_clean":"2024-03-09T00:06:06.076594+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:06:06.076594+0000","last_undegraded":"2024-03-09T00:06:06.076594+0000","last_fullsized":"2024-03-09T00:06:06.076594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081920+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:06:06.081920+0000","last_peered":"2024-03-09T00:06:06.081920+0000","last_clean":"2024-03-09T00:06:06.081920+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:06:06.081920+0000","last_undegraded":"2024-03-09T00:06:06.081920+0000","last_fullsized":"2024-03-09T00:06:06.081920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076436+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:06:06.076436+0000","last_peered":"2024-03-09T00:06:06.076436+0000","last_clean":"2024-03-09T00:06:06.076436+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:06:06.076436+0000","last_undegraded":"2024-03-09T00:06:06.076436+0000","last_fullsized":"2024-03-09T00:06:06.076436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079285+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:06:06.079285+0000","last_peered":"2024-03-09T00:06:06.079285+0000","last_clean":"2024-03-09T00:06:06.079285+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:06:06.079285+0000","last_undegraded":"2024-03-09T00:06:06.079285+0000","last_fullsized":"2024-03-09T00:06:06.079285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075921+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:06:06.075921+0000","last_peered":"2024-03-09T00:06:06.075921+0000","last_clean":"2024-03-09T00:06:06.075921+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:06:06.075921+0000","last_undegraded":"2024-03-09T00:06:06.075921+0000","last_fullsized":"2024-03-09T00:06:06.075921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079599+0000","last_change":"2024-03-09T00:05:05.059283+0000","last_active":"2024-03-09T00:06:06.079599+0000","last_peered":"2024-03-09T00:06:06.079599+0000","last_clean":"2024-03-09T00:06:06.079599+0000","last_became_active":"2024-03-09T00:05:05.058822+0000","last_became_peered":"2024-03-09T00:05:05.058822+0000","last_unstale":"2024-03-09T00:06:06.079599+0000","last_undegraded":"2024-03-09T00:06:06.079599+0000","last_fullsized":"2024-03-09T00:06:06.079599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:13:02.253810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276881+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:06:06.276881+0000","last_peered":"2024-03-09T00:06:06.276881+0000","last_clean":"2024-03-09T00:06:06.276881+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:06:06.276881+0000","last_undegraded":"2024-03-09T00:06:06.276881+0000","last_fullsized":"2024-03-09T00:06:06.276881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271661+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:06:06.271661+0000","last_peered":"2024-03-09T00:06:06.271661+0000","last_clean":"2024-03-09T00:06:06.271661+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:06:06.271661+0000","last_undegraded":"2024-03-09T00:06:06.271661+0000","last_fullsized":"2024-03-09T00:06:06.271661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079467+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:06:06.079467+0000","last_peered":"2024-03-09T00:06:06.079467+0000","last_clean":"2024-03-09T00:06:06.079467+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:06:06.079467+0000","last_undegraded":"2024-03-09T00:06:06.079467+0000","last_fullsized":"2024-03-09T00:06:06.079467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076789+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:06:06.076789+0000","last_peered":"2024-03-09T00:06:06.076789+0000","last_clean":"2024-03-09T00:06:06.076789+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:06:06.076789+0000","last_undegraded":"2024-03-09T00:06:06.076789+0000","last_fullsized":"2024-03-09T00:06:06.076789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.089907+0000","last_change":"2024-03-09T00:06:06.089420+0000","last_active":"2024-03-09T00:06:06.089907+0000","last_peered":"2024-03-09T00:06:06.089907+0000","last_clean":"2024-03-09T00:06:06.089907+0000","last_became_active":"2024-03-09T00:06:06.083164+0000","last_became_peered":"2024-03-09T00:06:06.083164+0000","last_unstale":"2024-03-09T00:06:06.089907+0000","last_undegraded":"2024-03-09T00:06:06.089907+0000","last_fullsized":"2024-03-09T00:06:06.089907+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:08:34.770550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075880+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:06:06.075880+0000","last_peered":"2024-03-09T00:06:06.075880+0000","last_clean":"2024-03-09T00:06:06.075880+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:06:06.075880+0000","last_undegraded":"2024-03-09T00:06:06.075880+0000","last_fullsized":"2024-03-09T00:06:06.075880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075786+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:06:06.075786+0000","last_peered":"2024-03-09T00:06:06.075786+0000","last_clean":"2024-03-09T00:06:06.075786+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:06:06.075786+0000","last_undegraded":"2024-03-09T00:06:06.075786+0000","last_fullsized":"2024-03-09T00:06:06.075786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082961+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:06:06.082961+0000","last_peered":"2024-03-09T00:06:06.082961+0000","last_clean":"2024-03-09T00:06:06.082961+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:06:06.082961+0000","last_undegraded":"2024-03-09T00:06:06.082961+0000","last_fullsized":"2024-03-09T00:06:06.082961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079668+0000","last_change":"2024-03-09T00:05:05.138598+0000","last_active":"2024-03-09T00:06:06.079668+0000","last_peered":"2024-03-09T00:06:06.079668+0000","last_clean":"2024-03-09T00:06:06.079668+0000","last_became_active":"2024-03-09T00:05:05.138166+0000","last_became_peered":"2024-03-09T00:05:05.138166+0000","last_unstale":"2024-03-09T00:06:06.079668+0000","last_undegraded":"2024-03-09T00:06:06.079668+0000","last_fullsized":"2024-03-09T00:06:06.079668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:59:42.864722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082189+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:06:06.082189+0000","last_peered":"2024-03-09T00:06:06.082189+0000","last_clean":"2024-03-09T00:06:06.082189+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:06:06.082189+0000","last_undegraded":"2024-03-09T00:06:06.082189+0000","last_fullsized":"2024-03-09T00:06:06.082189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276948+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:06:06.276948+0000","last_peered":"2024-03-09T00:06:06.276948+0000","last_clean":"2024-03-09T00:06:06.276948+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:06:06.276948+0000","last_undegraded":"2024-03-09T00:06:06.276948+0000","last_fullsized":"2024-03-09T00:06:06.276948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075133+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:06:06.075133+0000","last_peered":"2024-03-09T00:06:06.075133+0000","last_clean":"2024-03-09T00:06:06.075133+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:06:06.075133+0000","last_undegraded":"2024-03-09T00:06:06.075133+0000","last_fullsized":"2024-03-09T00:06:06.075133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076732+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:06:06.076732+0000","last_peered":"2024-03-09T00:06:06.076732+0000","last_clean":"2024-03-09T00:06:06.076732+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:06:06.076732+0000","last_undegraded":"2024-03-09T00:06:06.076732+0000","last_fullsized":"2024-03-09T00:06:06.076732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079746+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:06:06.079746+0000","last_peered":"2024-03-09T00:06:06.079746+0000","last_clean":"2024-03-09T00:06:06.079746+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:06:06.079746+0000","last_undegraded":"2024-03-09T00:06:06.079746+0000","last_fullsized":"2024-03-09T00:06:06.079746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075373+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:06:06.075373+0000","last_peered":"2024-03-09T00:06:06.075373+0000","last_clean":"2024-03-09T00:06:06.075373+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:06:06.075373+0000","last_undegraded":"2024-03-09T00:06:06.075373+0000","last_fullsized":"2024-03-09T00:06:06.075373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076899+0000","last_change":"2024-03-09T00:05:05.138164+0000","last_active":"2024-03-09T00:06:06.076899+0000","last_peered":"2024-03-09T00:06:06.076899+0000","last_clean":"2024-03-09T00:06:06.076899+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.076899+0000","last_undegraded":"2024-03-09T00:06:06.076899+0000","last_fullsized":"2024-03-09T00:06:06.076899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:12:44.310276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076054+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:06:06.076054+0000","last_peered":"2024-03-09T00:06:06.076054+0000","last_clean":"2024-03-09T00:06:06.076054+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.076054+0000","last_undegraded":"2024-03-09T00:06:06.076054+0000","last_fullsized":"2024-03-09T00:06:06.076054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076336+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:06:06.076336+0000","last_peered":"2024-03-09T00:06:06.076336+0000","last_clean":"2024-03-09T00:06:06.076336+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:06:06.076336+0000","last_undegraded":"2024-03-09T00:06:06.076336+0000","last_fullsized":"2024-03-09T00:06:06.076336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079810+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:06:06.079810+0000","last_peered":"2024-03-09T00:06:06.079810+0000","last_clean":"2024-03-09T00:06:06.079810+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:06:06.079810+0000","last_undegraded":"2024-03-09T00:06:06.079810+0000","last_fullsized":"2024-03-09T00:06:06.079810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276945+0000","last_change":"2024-03-09T00:05:05.058253+0000","last_active":"2024-03-09T00:06:06.276945+0000","last_peered":"2024-03-09T00:06:06.276945+0000","last_clean":"2024-03-09T00:06:06.276945+0000","last_became_active":"2024-03-09T00:05:05.057864+0000","last_became_peered":"2024-03-09T00:05:05.057864+0000","last_unstale":"2024-03-09T00:06:06.276945+0000","last_undegraded":"2024-03-09T00:06:06.276945+0000","last_fullsized":"2024-03-09T00:06:06.276945+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:33:14.993355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271946+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:06:06.271946+0000","last_peered":"2024-03-09T00:06:06.271946+0000","last_clean":"2024-03-09T00:06:06.271946+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:06:06.271946+0000","last_undegraded":"2024-03-09T00:06:06.271946+0000","last_fullsized":"2024-03-09T00:06:06.271946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079321+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:06:06.079321+0000","last_peered":"2024-03-09T00:06:06.079321+0000","last_clean":"2024-03-09T00:06:06.079321+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:06:06.079321+0000","last_undegraded":"2024-03-09T00:06:06.079321+0000","last_fullsized":"2024-03-09T00:06:06.079321+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076429+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:06:06.076429+0000","last_peered":"2024-03-09T00:06:06.076429+0000","last_clean":"2024-03-09T00:06:06.076429+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:06:06.076429+0000","last_undegraded":"2024-03-09T00:06:06.076429+0000","last_fullsized":"2024-03-09T00:06:06.076429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076267+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:06:06.076267+0000","last_peered":"2024-03-09T00:06:06.076267+0000","last_clean":"2024-03-09T00:06:06.076267+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:06:06.076267+0000","last_undegraded":"2024-03-09T00:06:06.076267+0000","last_fullsized":"2024-03-09T00:06:06.076267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079600+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:06:06.079600+0000","last_peered":"2024-03-09T00:06:06.079600+0000","last_clean":"2024-03-09T00:06:06.079600+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:06:06.079600+0000","last_undegraded":"2024-03-09T00:06:06.079600+0000","last_fullsized":"2024-03-09T00:06:06.079600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079335+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:06:06.079335+0000","last_peered":"2024-03-09T00:06:06.079335+0000","last_clean":"2024-03-09T00:06:06.079335+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:06:06.079335+0000","last_undegraded":"2024-03-09T00:06:06.079335+0000","last_fullsized":"2024-03-09T00:06:06.079335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.278451+0000","last_change":"2024-03-09T00:06:06.278068+0000","last_active":"2024-03-09T00:06:06.278451+0000","last_peered":"2024-03-09T00:06:06.278451+0000","last_clean":"2024-03-09T00:06:06.278451+0000","last_became_active":"2024-03-09T00:06:06.277772+0000","last_became_peered":"2024-03-09T00:06:06.277772+0000","last_unstale":"2024-03-09T00:06:06.278451+0000","last_undegraded":"2024-03-09T00:06:06.278451+0000","last_fullsized":"2024-03-09T00:06:06.278451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:36:33.637238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076841+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:06:06.076841+0000","last_peered":"2024-03-09T00:06:06.076841+0000","last_clean":"2024-03-09T00:06:06.076841+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:06:06.076841+0000","last_undegraded":"2024-03-09T00:06:06.076841+0000","last_fullsized":"2024-03-09T00:06:06.076841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921495+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:06:06.921495+0000","last_peered":"2024-03-09T00:06:06.921495+0000","last_clean":"2024-03-09T00:06:06.921495+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:06:06.921495+0000","last_undegraded":"2024-03-09T00:06:06.921495+0000","last_fullsized":"2024-03-09T00:06:06.921495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'8","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:23.979195+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:06:23.979195+0000","last_peered":"2024-03-09T00:06:23.979195+0000","last_clean":"2024-03-09T00:06:23.979195+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:06:23.979195+0000","last_undegraded":"2024-03-09T00:06:23.979195+0000","last_fullsized":"2024-03-09T00:06:23.979195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921938+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:06:06.921938+0000","last_peered":"2024-03-09T00:06:06.921938+0000","last_clean":"2024-03-09T00:06:06.921938+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:06:06.921938+0000","last_undegraded":"2024-03-09T00:06:06.921938+0000","last_fullsized":"2024-03-09T00:06:06.921938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081216+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:06:06.081216+0000","last_peered":"2024-03-09T00:06:06.081216+0000","last_clean":"2024-03-09T00:06:06.081216+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:06:06.081216+0000","last_undegraded":"2024-03-09T00:06:06.081216+0000","last_fullsized":"2024-03-09T00:06:06.081216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272193+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:06:06.272193+0000","last_peered":"2024-03-09T00:06:06.272193+0000","last_clean":"2024-03-09T00:06:06.272193+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:06:06.272193+0000","last_undegraded":"2024-03-09T00:06:06.272193+0000","last_fullsized":"2024-03-09T00:06:06.272193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081386+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:06:06.081386+0000","last_peered":"2024-03-09T00:06:06.081386+0000","last_clean":"2024-03-09T00:06:06.081386+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:06:06.081386+0000","last_undegraded":"2024-03-09T00:06:06.081386+0000","last_fullsized":"2024-03-09T00:06:06.081386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271932+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:06:06.271932+0000","last_peered":"2024-03-09T00:06:06.271932+0000","last_clean":"2024-03-09T00:06:06.271932+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:06:06.271932+0000","last_undegraded":"2024-03-09T00:06:06.271932+0000","last_fullsized":"2024-03-09T00:06:06.271932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513561+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:06:06.513561+0000","last_peered":"2024-03-09T00:06:06.513561+0000","last_clean":"2024-03-09T00:06:06.513561+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:06:06.513561+0000","last_undegraded":"2024-03-09T00:06:06.513561+0000","last_fullsized":"2024-03-09T00:06:06.513561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081820+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:06:06.081820+0000","last_peered":"2024-03-09T00:06:06.081820+0000","last_clean":"2024-03-09T00:06:06.081820+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:06:06.081820+0000","last_undegraded":"2024-03-09T00:06:06.081820+0000","last_fullsized":"2024-03-09T00:06:06.081820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277174+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:06:06.277174+0000","last_peered":"2024-03-09T00:06:06.277174+0000","last_clean":"2024-03-09T00:06:06.277174+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:06:06.277174+0000","last_undegraded":"2024-03-09T00:06:06.277174+0000","last_fullsized":"2024-03-09T00:06:06.277174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921579+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:06:06.921579+0000","last_peered":"2024-03-09T00:06:06.921579+0000","last_clean":"2024-03-09T00:06:06.921579+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:06:06.921579+0000","last_undegraded":"2024-03-09T00:06:06.921579+0000","last_fullsized":"2024-03-09T00:06:06.921579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079350+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:06:06.079350+0000","last_peered":"2024-03-09T00:06:06.079350+0000","last_clean":"2024-03-09T00:06:06.079350+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:06:06.079350+0000","last_undegraded":"2024-03-09T00:06:06.079350+0000","last_fullsized":"2024-03-09T00:06:06.079350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271971+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:06:06.271971+0000","last_peered":"2024-03-09T00:06:06.271971+0000","last_clean":"2024-03-09T00:06:06.271971+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:06:06.271971+0000","last_undegraded":"2024-03-09T00:06:06.271971+0000","last_fullsized":"2024-03-09T00:06:06.271971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081894+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:06:06.081894+0000","last_peered":"2024-03-09T00:06:06.081894+0000","last_clean":"2024-03-09T00:06:06.081894+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:06:06.081894+0000","last_undegraded":"2024-03-09T00:06:06.081894+0000","last_fullsized":"2024-03-09T00:06:06.081894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921869+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:06:06.921869+0000","last_peered":"2024-03-09T00:06:06.921869+0000","last_clean":"2024-03-09T00:06:06.921869+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:06:06.921869+0000","last_undegraded":"2024-03-09T00:06:06.921869+0000","last_fullsized":"2024-03-09T00:06:06.921869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271986+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:06:06.271986+0000","last_peered":"2024-03-09T00:06:06.271986+0000","last_clean":"2024-03-09T00:06:06.271986+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:06:06.271986+0000","last_undegraded":"2024-03-09T00:06:06.271986+0000","last_fullsized":"2024-03-09T00:06:06.271986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082328+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:06:06.082328+0000","last_peered":"2024-03-09T00:06:06.082328+0000","last_clean":"2024-03-09T00:06:06.082328+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:06:06.082328+0000","last_undegraded":"2024-03-09T00:06:06.082328+0000","last_fullsized":"2024-03-09T00:06:06.082328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079588+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:06:06.079588+0000","last_peered":"2024-03-09T00:06:06.079588+0000","last_clean":"2024-03-09T00:06:06.079588+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:06:06.079588+0000","last_undegraded":"2024-03-09T00:06:06.079588+0000","last_fullsized":"2024-03-09T00:06:06.079588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513345+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:06:06.513345+0000","last_peered":"2024-03-09T00:06:06.513345+0000","last_clean":"2024-03-09T00:06:06.513345+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:06:06.513345+0000","last_undegraded":"2024-03-09T00:06:06.513345+0000","last_fullsized":"2024-03-09T00:06:06.513345+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081834+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:06:06.081834+0000","last_peered":"2024-03-09T00:06:06.081834+0000","last_clean":"2024-03-09T00:06:06.081834+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:06:06.081834+0000","last_undegraded":"2024-03-09T00:06:06.081834+0000","last_fullsized":"2024-03-09T00:06:06.081834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079996+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:06:06.079996+0000","last_peered":"2024-03-09T00:06:06.079996+0000","last_clean":"2024-03-09T00:06:06.079996+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:06:06.079996+0000","last_undegraded":"2024-03-09T00:06:06.079996+0000","last_fullsized":"2024-03-09T00:06:06.079996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081211+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:06:06.081211+0000","last_peered":"2024-03-09T00:06:06.081211+0000","last_clean":"2024-03-09T00:06:06.081211+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:06:06.081211+0000","last_undegraded":"2024-03-09T00:06:06.081211+0000","last_fullsized":"2024-03-09T00:06:06.081211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076463+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:06:06.076463+0000","last_peered":"2024-03-09T00:06:06.076463+0000","last_clean":"2024-03-09T00:06:06.076463+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:06:06.076463+0000","last_undegraded":"2024-03-09T00:06:06.076463+0000","last_fullsized":"2024-03-09T00:06:06.076463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079730+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:06:06.079730+0000","last_peered":"2024-03-09T00:06:06.079730+0000","last_clean":"2024-03-09T00:06:06.079730+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:06:06.079730+0000","last_undegraded":"2024-03-09T00:06:06.079730+0000","last_fullsized":"2024-03-09T00:06:06.079730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075532+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:06:06.075532+0000","last_peered":"2024-03-09T00:06:06.075532+0000","last_clean":"2024-03-09T00:06:06.075532+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:06:06.075532+0000","last_undegraded":"2024-03-09T00:06:06.075532+0000","last_fullsized":"2024-03-09T00:06:06.075532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076859+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:06:06.076859+0000","last_peered":"2024-03-09T00:06:06.076859+0000","last_clean":"2024-03-09T00:06:06.076859+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:06:06.076859+0000","last_undegraded":"2024-03-09T00:06:06.076859+0000","last_fullsized":"2024-03-09T00:06:06.076859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080006+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:06:06.080006+0000","last_peered":"2024-03-09T00:06:06.080006+0000","last_clean":"2024-03-09T00:06:06.080006+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:06:06.080006+0000","last_undegraded":"2024-03-09T00:06:06.080006+0000","last_fullsized":"2024-03-09T00:06:06.080006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079554+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:06:06.079554+0000","last_peered":"2024-03-09T00:06:06.079554+0000","last_clean":"2024-03-09T00:06:06.079554+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:06:06.079554+0000","last_undegraded":"2024-03-09T00:06:06.079554+0000","last_fullsized":"2024-03-09T00:06:06.079554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081561+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:06:06.081561+0000","last_peered":"2024-03-09T00:06:06.081561+0000","last_clean":"2024-03-09T00:06:06.081561+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:06:06.081561+0000","last_undegraded":"2024-03-09T00:06:06.081561+0000","last_fullsized":"2024-03-09T00:06:06.081561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081789+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:06:06.081789+0000","last_peered":"2024-03-09T00:06:06.081789+0000","last_clean":"2024-03-09T00:06:06.081789+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:06:06.081789+0000","last_undegraded":"2024-03-09T00:06:06.081789+0000","last_fullsized":"2024-03-09T00:06:06.081789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081489+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:06:06.081489+0000","last_peered":"2024-03-09T00:06:06.081489+0000","last_clean":"2024-03-09T00:06:06.081489+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:06:06.081489+0000","last_undegraded":"2024-03-09T00:06:06.081489+0000","last_fullsized":"2024-03-09T00:06:06.081489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076818+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:06:06.076818+0000","last_peered":"2024-03-09T00:06:06.076818+0000","last_clean":"2024-03-09T00:06:06.076818+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:06:06.076818+0000","last_undegraded":"2024-03-09T00:06:06.076818+0000","last_fullsized":"2024-03-09T00:06:06.076818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079815+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:06:06.079815+0000","last_peered":"2024-03-09T00:06:06.079815+0000","last_clean":"2024-03-09T00:06:06.079815+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:06:06.079815+0000","last_undegraded":"2024-03-09T00:06:06.079815+0000","last_fullsized":"2024-03-09T00:06:06.079815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082078+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:06:06.082078+0000","last_peered":"2024-03-09T00:06:06.082078+0000","last_clean":"2024-03-09T00:06:06.082078+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:06:06.082078+0000","last_undegraded":"2024-03-09T00:06:06.082078+0000","last_fullsized":"2024-03-09T00:06:06.082078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921924+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:06:06.921924+0000","last_peered":"2024-03-09T00:06:06.921924+0000","last_clean":"2024-03-09T00:06:06.921924+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:06:06.921924+0000","last_undegraded":"2024-03-09T00:06:06.921924+0000","last_fullsized":"2024-03-09T00:06:06.921924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076018+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:06:06.076018+0000","last_peered":"2024-03-09T00:06:06.076018+0000","last_clean":"2024-03-09T00:06:06.076018+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:06:06.076018+0000","last_undegraded":"2024-03-09T00:06:06.076018+0000","last_fullsized":"2024-03-09T00:06:06.076018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921769+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:06:06.921769+0000","last_peered":"2024-03-09T00:06:06.921769+0000","last_clean":"2024-03-09T00:06:06.921769+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:06:06.921769+0000","last_undegraded":"2024-03-09T00:06:06.921769+0000","last_fullsized":"2024-03-09T00:06:06.921769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513579+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:06:06.513579+0000","last_peered":"2024-03-09T00:06:06.513579+0000","last_clean":"2024-03-09T00:06:06.513579+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:06:06.513579+0000","last_undegraded":"2024-03-09T00:06:06.513579+0000","last_fullsized":"2024-03-09T00:06:06.513579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076412+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:06:06.076412+0000","last_peered":"2024-03-09T00:06:06.076412+0000","last_clean":"2024-03-09T00:06:06.076412+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:06:06.076412+0000","last_undegraded":"2024-03-09T00:06:06.076412+0000","last_fullsized":"2024-03-09T00:06:06.076412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079497+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:06:06.079497+0000","last_peered":"2024-03-09T00:06:06.079497+0000","last_clean":"2024-03-09T00:06:06.079497+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:06:06.079497+0000","last_undegraded":"2024-03-09T00:06:06.079497+0000","last_fullsized":"2024-03-09T00:06:06.079497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70084,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":258,"num_read_kb":200,"num_write":89,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":199198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":519110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32868,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712284,"statfs":{"total":95995035648,"available":95961378816,"internally_reserved":0,"allocated":1732608,"data_stored":1174862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":523788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":519714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1114156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":515014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":578927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":515104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":578653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":515118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":519132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1658880,"data_stored":1109660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:06:40.900 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-09T00:06:40.901 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-09T00:06:41.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 -- 172.21.15.73:0/3265337711 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8102eb0 msgr2=0x7fe3a8109740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/3265337711 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8102eb0 0x7fe3a8109740 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 crypto rx=0x7fe398009900 tx=0x7fe39802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:41.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 -- 172.21.15.73:0/3265337711 shutdown_connections 2024-03-09T00:06:41.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/3265337711 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8102eb0 0x7fe3a8109740 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/3265337711 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/3265337711 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8101300 0x7fe3a8101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 -- 172.21.15.73:0/3265337711 >> 172.21.15.73:0/3265337711 conn(0x7fe3a80fc8d0 msgr2=0x7fe3a80fecf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 -- 172.21.15.73:0/3265337711 shutdown_connections 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.008+0000 7fe3b00e8640 1 -- 172.21.15.73:0/3265337711 wait complete. 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 Processor -- start 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 -- start start 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8197b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8102eb0 0x7fe3a819ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810cfe0 con 0x7fe3a8101300 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810ce60 con 0x7fe3a8102eb0 2024-03-09T00:06:41.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3b00e8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3a810d160 con 0x7fe3a81024f0 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60228/0 (socket says 172.21.15.73:60228) 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 -- 172.21.15.73:0/4046049790 learned_addr learned my addr 172.21.15.73:0/4046049790 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ae685640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8102eb0 0x7fe3a819ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ad683640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8197b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 -- 172.21.15.73:0/4046049790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 msgr2=0x7fe3a8197b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8197b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 -- 172.21.15.73:0/4046049790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8102eb0 msgr2=0x7fe3a819ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8102eb0 0x7fe3a819ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 -- 172.21.15.73:0/4046049790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3a819f240 con 0x7fe3a8101300 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ad683640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3ade84640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000e9c0 tx=0x7fe3a000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.009+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a000cd80 con 0x7fe3a8101300 2024-03-09T00:06:41.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.010+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a819f530 con 0x7fe3a8101300 2024-03-09T00:06:41.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.010+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a00035f0 con 0x7fe3a8101300 2024-03-09T00:06:41.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.010+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0010690 con 0x7fe3a8101300 2024-03-09T00:06:41.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.010+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3a819fad0 con 0x7fe3a8101300 2024-03-09T00:06:41.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.011+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3a81085c0 con 0x7fe3a8101300 2024-03-09T00:06:41.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.012+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fe3a00037a0 con 0x7fe3a8101300 2024-03-09T00:06:41.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.012+0000 7fe3a6ffd640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.012+0000 7fe3ad683640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.012+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fe3a00cf270 con 0x7fe3a8101300 2024-03-09T00:06:41.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.013+0000 7fe3ad683640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 0x7fe3900a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8198b30 tx=0x7fe39c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.016+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe3a009bf60 con 0x7fe3a8101300 2024-03-09T00:06:41.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:41 smithi073 ceph-mon[32309]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-09T00:06:41.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:41 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1998829124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:41.186 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.191+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fe3a8061730 con 0x7fe3900a2800 2024-03-09T00:06:41.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.200+0000 7fe3a6ffd640 1 -- 172.21.15.73:0/4046049790 <== mgr.14152 v2:172.21.15.73:6800/2701889057 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+340392 (secure 0 0 0) 0x7fe3a8061730 con 0x7fe3900a2800 2024-03-09T00:06:41.199 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:41.201 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-03-09T00:06:41.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 msgr2=0x7fe3900a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 0x7fe3900a4cc0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fe3a8198b30 tx=0x7fe39c009500 comp rx=0 tx=0).stop 2024-03-09T00:06:41.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 msgr2=0x7fe3a8197620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fe3a000e9c0 tx=0x7fe3a000ee80 comp rx=0 tx=0).stop 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 shutdown_connections 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fe3900a2800 0x7fe3900a4cc0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe3a8102eb0 0x7fe3a819ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe3a81024f0 0x7fe3a8197b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 --2- 172.21.15.73:0/4046049790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe3a8101300 0x7fe3a8197620 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 >> 172.21.15.73:0/4046049790 conn(0x7fe3a80fc8d0 msgr2=0x7fe3a8105790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 shutdown_connections 2024-03-09T00:06:41.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.204+0000 7fe3b00e8640 1 -- 172.21.15.73:0/4046049790 wait complete. 2024-03-09T00:06:41.210 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2024-03-09T00:06:39.967346+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660686,"num_objects":26,"num_object_clones":0,"num_object_copies":80,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":26,"num_whiteouts":0,"num_read":394,"num_read_kb":319,"num_write":272,"num_write_kb":3042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":715,"ondisk_log_size":715,"up":419,"acting":419,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":419,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":396832,"kb_used_data":14340,"kb_used_omap":18,"kb_used_meta":382317,"kb_avail":1124544992,"statfs":{"total":1151940427776,"available":1151534071808,"internally_reserved":0,"allocated":14684160,"data_stored":8183238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":19143,"internal_metadata":391492921},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":5681,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":32,"num_read_kb":22,"num_write":5,"num_write_kb":9,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002891"},"pg_stats":[{"pgid":"3.3f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079857+0000","last_change":"2024-03-09T00:04:13.933188+0000","last_active":"2024-03-09T00:06:06.079857+0000","last_peered":"2024-03-09T00:06:06.079857+0000","last_clean":"2024-03-09T00:06:06.079857+0000","last_became_active":"2024-03-09T00:04:13.932745+0000","last_became_peered":"2024-03-09T00:04:13.932745+0000","last_unstale":"2024-03-09T00:06:06.079857+0000","last_undegraded":"2024-03-09T00:06:06.079857+0000","last_fullsized":"2024-03-09T00:06:06.079857+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:52.568291+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081264+0000","last_change":"2024-03-09T00:04:13.930507+0000","last_active":"2024-03-09T00:06:06.081264+0000","last_peered":"2024-03-09T00:06:06.081264+0000","last_clean":"2024-03-09T00:06:06.081264+0000","last_became_active":"2024-03-09T00:04:13.930163+0000","last_became_peered":"2024-03-09T00:04:13.930163+0000","last_unstale":"2024-03-09T00:06:06.081264+0000","last_undegraded":"2024-03-09T00:06:06.081264+0000","last_fullsized":"2024-03-09T00:06:06.081264+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:37.006913+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081167+0000","last_change":"2024-03-09T00:04:13.931623+0000","last_active":"2024-03-09T00:06:06.081167+0000","last_peered":"2024-03-09T00:06:06.081167+0000","last_clean":"2024-03-09T00:06:06.081167+0000","last_became_active":"2024-03-09T00:04:13.931314+0000","last_became_peered":"2024-03-09T00:04:13.931314+0000","last_unstale":"2024-03-09T00:06:06.081167+0000","last_undegraded":"2024-03-09T00:06:06.081167+0000","last_fullsized":"2024-03-09T00:06:06.081167+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:19:30.347294+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272147+0000","last_change":"2024-03-09T00:04:13.940210+0000","last_active":"2024-03-09T00:06:06.272147+0000","last_peered":"2024-03-09T00:06:06.272147+0000","last_clean":"2024-03-09T00:06:06.272147+0000","last_became_active":"2024-03-09T00:04:13.939890+0000","last_became_peered":"2024-03-09T00:04:13.939890+0000","last_unstale":"2024-03-09T00:06:06.272147+0000","last_undegraded":"2024-03-09T00:06:06.272147+0000","last_fullsized":"2024-03-09T00:06:06.272147+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:32:58.258406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081462+0000","last_change":"2024-03-09T00:04:13.924984+0000","last_active":"2024-03-09T00:06:06.081462+0000","last_peered":"2024-03-09T00:06:06.081462+0000","last_clean":"2024-03-09T00:06:06.081462+0000","last_became_active":"2024-03-09T00:04:13.924652+0000","last_became_peered":"2024-03-09T00:04:13.924652+0000","last_unstale":"2024-03-09T00:06:06.081462+0000","last_undegraded":"2024-03-09T00:06:06.081462+0000","last_fullsized":"2024-03-09T00:06:06.081462+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:58:01.092379+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"91'6","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075196+0000","last_change":"2024-03-09T00:05:05.058033+0000","last_active":"2024-03-09T00:06:06.075196+0000","last_peered":"2024-03-09T00:06:06.075196+0000","last_clean":"2024-03-09T00:06:06.075196+0000","last_became_active":"2024-03-09T00:05:05.057575+0000","last_became_peered":"2024-03-09T00:05:05.057575+0000","last_unstale":"2024-03-09T00:06:06.075196+0000","last_undegraded":"2024-03-09T00:06:06.075196+0000","last_fullsized":"2024-03-09T00:06:06.075196+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:35:58.842128+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,8],"acting":[10,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"91'5","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079431+0000","last_change":"2024-03-09T00:04:13.931503+0000","last_active":"2024-03-09T00:06:06.079431+0000","last_peered":"2024-03-09T00:06:06.079431+0000","last_clean":"2024-03-09T00:06:06.079431+0000","last_became_active":"2024-03-09T00:04:13.931166+0000","last_became_peered":"2024-03-09T00:04:13.931166+0000","last_unstale":"2024-03-09T00:06:06.079431+0000","last_undegraded":"2024-03-09T00:06:06.079431+0000","last_fullsized":"2024-03-09T00:06:06.079431+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:57:14.576027+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"91'9","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921426+0000","last_change":"2024-03-09T00:04:13.932928+0000","last_active":"2024-03-09T00:06:06.921426+0000","last_peered":"2024-03-09T00:06:06.921426+0000","last_clean":"2024-03-09T00:06:06.921426+0000","last_became_active":"2024-03-09T00:04:13.932596+0000","last_became_peered":"2024-03-09T00:04:13.932596+0000","last_unstale":"2024-03-09T00:06:06.921426+0000","last_undegraded":"2024-03-09T00:06:06.921426+0000","last_fullsized":"2024-03-09T00:06:06.921426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:59:33.360654+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513955+0000","last_change":"2024-03-09T00:04:13.931142+0000","last_active":"2024-03-09T00:06:06.513955+0000","last_peered":"2024-03-09T00:06:06.513955+0000","last_clean":"2024-03-09T00:06:06.513955+0000","last_became_active":"2024-03-09T00:04:13.930826+0000","last_became_peered":"2024-03-09T00:04:13.930826+0000","last_unstale":"2024-03-09T00:06:06.513955+0000","last_undegraded":"2024-03-09T00:06:06.513955+0000","last_fullsized":"2024-03-09T00:06:06.513955+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:36.787782+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"91'16","reported_seq":46,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075525+0000","last_change":"2024-03-09T00:04:13.932707+0000","last_active":"2024-03-09T00:06:06.075525+0000","last_peered":"2024-03-09T00:06:06.075525+0000","last_clean":"2024-03-09T00:06:06.075525+0000","last_became_active":"2024-03-09T00:04:13.932397+0000","last_became_peered":"2024-03-09T00:04:13.932397+0000","last_unstale":"2024-03-09T00:06:06.075525+0000","last_undegraded":"2024-03-09T00:06:06.075525+0000","last_fullsized":"2024-03-09T00:06:06.075525+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:51:08.310512+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"91'6","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079896+0000","last_change":"2024-03-09T00:04:13.937032+0000","last_active":"2024-03-09T00:06:06.079896+0000","last_peered":"2024-03-09T00:06:06.079896+0000","last_clean":"2024-03-09T00:06:06.079896+0000","last_became_active":"2024-03-09T00:04:13.934554+0000","last_became_peered":"2024-03-09T00:04:13.934554+0000","last_unstale":"2024-03-09T00:06:06.079896+0000","last_undegraded":"2024-03-09T00:06:06.079896+0000","last_fullsized":"2024-03-09T00:06:06.079896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:18:50.169810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"91'7","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271903+0000","last_change":"2024-03-09T00:04:13.940301+0000","last_active":"2024-03-09T00:06:06.271903+0000","last_peered":"2024-03-09T00:06:06.271903+0000","last_clean":"2024-03-09T00:06:06.271903+0000","last_became_active":"2024-03-09T00:04:13.939975+0000","last_became_peered":"2024-03-09T00:04:13.939975+0000","last_unstale":"2024-03-09T00:06:06.271903+0000","last_undegraded":"2024-03-09T00:06:06.271903+0000","last_fullsized":"2024-03-09T00:06:06.271903+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:28:50.643597+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"91'10","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076426+0000","last_change":"2024-03-09T00:04:13.928891+0000","last_active":"2024-03-09T00:06:06.076426+0000","last_peered":"2024-03-09T00:06:06.076426+0000","last_clean":"2024-03-09T00:06:06.076426+0000","last_became_active":"2024-03-09T00:04:13.928610+0000","last_became_peered":"2024-03-09T00:04:13.928610+0000","last_unstale":"2024-03-09T00:06:06.076426+0000","last_undegraded":"2024-03-09T00:06:06.076426+0000","last_fullsized":"2024-03-09T00:06:06.076426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:32:31.265903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081033+0000","last_change":"2024-03-09T00:04:13.925362+0000","last_active":"2024-03-09T00:06:06.081033+0000","last_peered":"2024-03-09T00:06:06.081033+0000","last_clean":"2024-03-09T00:06:06.081033+0000","last_became_active":"2024-03-09T00:04:13.925063+0000","last_became_peered":"2024-03-09T00:04:13.925063+0000","last_unstale":"2024-03-09T00:06:06.081033+0000","last_undegraded":"2024-03-09T00:06:06.081033+0000","last_fullsized":"2024-03-09T00:06:06.081033+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:55.257443+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080896+0000","last_change":"2024-03-09T00:04:13.934862+0000","last_active":"2024-03-09T00:06:06.080896+0000","last_peered":"2024-03-09T00:06:06.080896+0000","last_clean":"2024-03-09T00:06:06.080896+0000","last_became_active":"2024-03-09T00:04:13.934521+0000","last_became_peered":"2024-03-09T00:04:13.934521+0000","last_unstale":"2024-03-09T00:06:06.080896+0000","last_undegraded":"2024-03-09T00:06:06.080896+0000","last_fullsized":"2024-03-09T00:06:06.080896+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:41:21.027549+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"91'9","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513424+0000","last_change":"2024-03-09T00:04:13.927689+0000","last_active":"2024-03-09T00:06:06.513424+0000","last_peered":"2024-03-09T00:06:06.513424+0000","last_clean":"2024-03-09T00:06:06.513424+0000","last_became_active":"2024-03-09T00:04:13.927148+0000","last_became_peered":"2024-03-09T00:04:13.927148+0000","last_unstale":"2024-03-09T00:06:06.513424+0000","last_undegraded":"2024-03-09T00:06:06.513424+0000","last_fullsized":"2024-03-09T00:06:06.513424+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:18:36.575815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079620+0000","last_change":"2024-03-09T00:04:13.934360+0000","last_active":"2024-03-09T00:06:06.079620+0000","last_peered":"2024-03-09T00:06:06.079620+0000","last_clean":"2024-03-09T00:06:06.079620+0000","last_became_active":"2024-03-09T00:04:13.933987+0000","last_became_peered":"2024-03-09T00:04:13.933987+0000","last_unstale":"2024-03-09T00:06:06.079620+0000","last_undegraded":"2024-03-09T00:06:06.079620+0000","last_fullsized":"2024-03-09T00:06:06.079620+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:14:14.149755+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277255+0000","last_change":"2024-03-09T00:04:13.930005+0000","last_active":"2024-03-09T00:06:06.277255+0000","last_peered":"2024-03-09T00:06:06.277255+0000","last_clean":"2024-03-09T00:06:06.277255+0000","last_became_active":"2024-03-09T00:04:13.929662+0000","last_became_peered":"2024-03-09T00:04:13.929662+0000","last_unstale":"2024-03-09T00:06:06.277255+0000","last_undegraded":"2024-03-09T00:06:06.277255+0000","last_fullsized":"2024-03-09T00:06:06.277255+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:25:23.972787+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081723+0000","last_change":"2024-03-09T00:04:13.930014+0000","last_active":"2024-03-09T00:06:06.081723+0000","last_peered":"2024-03-09T00:06:06.081723+0000","last_clean":"2024-03-09T00:06:06.081723+0000","last_became_active":"2024-03-09T00:04:13.929686+0000","last_became_peered":"2024-03-09T00:04:13.929686+0000","last_unstale":"2024-03-09T00:06:06.081723+0000","last_undegraded":"2024-03-09T00:06:06.081723+0000","last_fullsized":"2024-03-09T00:06:06.081723+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:37.604335+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"4.19","version":"0'0","reported_seq":29,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081285+0000","last_change":"2024-03-09T00:05:05.138220+0000","last_active":"2024-03-09T00:06:06.081285+0000","last_peered":"2024-03-09T00:06:06.081285+0000","last_clean":"2024-03-09T00:06:06.081285+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.081285+0000","last_undegraded":"2024-03-09T00:06:06.081285+0000","last_fullsized":"2024-03-09T00:06:06.081285+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:59:41.455601+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8,0],"acting":[2,3,8,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"84'4","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277194+0000","last_change":"2024-03-09T00:04:13.333234+0000","last_active":"2024-03-09T00:06:06.277194+0000","last_peered":"2024-03-09T00:06:06.277194+0000","last_clean":"2024-03-09T00:06:06.277194+0000","last_became_active":"2024-03-09T00:04:13.332876+0000","last_became_peered":"2024-03-09T00:04:13.332876+0000","last_unstale":"2024-03-09T00:06:06.277194+0000","last_undegraded":"2024-03-09T00:06:06.277194+0000","last_fullsized":"2024-03-09T00:06:06.277194+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:26:09.368200+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922002+0000","last_change":"2024-03-09T00:04:13.933265+0000","last_active":"2024-03-09T00:06:06.922002+0000","last_peered":"2024-03-09T00:06:06.922002+0000","last_clean":"2024-03-09T00:06:06.922002+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.922002+0000","last_undegraded":"2024-03-09T00:06:06.922002+0000","last_fullsized":"2024-03-09T00:06:06.922002+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:01:53.276658+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.18","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271466+0000","last_change":"2024-03-09T00:04:17.957543+0000","last_active":"2024-03-09T00:06:06.271466+0000","last_peered":"2024-03-09T00:06:06.271466+0000","last_clean":"2024-03-09T00:06:06.271466+0000","last_became_active":"2024-03-09T00:04:17.956844+0000","last_became_peered":"2024-03-09T00:04:17.956844+0000","last_unstale":"2024-03-09T00:06:06.271466+0000","last_undegraded":"2024-03-09T00:06:06.271466+0000","last_fullsized":"2024-03-09T00:06:06.271466+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:35:04.744121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,9,3],"acting":[4,6,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1e","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.922049+0000","last_change":"2024-03-09T00:04:13.337299+0000","last_active":"2024-03-09T00:06:06.922049+0000","last_peered":"2024-03-09T00:06:06.922049+0000","last_clean":"2024-03-09T00:06:06.922049+0000","last_became_active":"2024-03-09T00:04:13.336965+0000","last_became_peered":"2024-03-09T00:04:13.336965+0000","last_unstale":"2024-03-09T00:06:06.922049+0000","last_undegraded":"2024-03-09T00:06:06.922049+0000","last_fullsized":"2024-03-09T00:06:06.922049+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:34.637716+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081306+0000","last_change":"2024-03-09T00:04:13.932323+0000","last_active":"2024-03-09T00:06:06.081306+0000","last_peered":"2024-03-09T00:06:06.081306+0000","last_clean":"2024-03-09T00:06:06.081306+0000","last_became_active":"2024-03-09T00:04:13.931716+0000","last_became_peered":"2024-03-09T00:04:13.931716+0000","last_unstale":"2024-03-09T00:06:06.081306+0000","last_undegraded":"2024-03-09T00:06:06.081306+0000","last_fullsized":"2024-03-09T00:06:06.081306+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:38:36.664934+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.1b","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271674+0000","last_change":"2024-03-09T00:04:17.960487+0000","last_active":"2024-03-09T00:06:06.271674+0000","last_peered":"2024-03-09T00:06:06.271674+0000","last_clean":"2024-03-09T00:06:06.271674+0000","last_became_active":"2024-03-09T00:04:17.959940+0000","last_became_peered":"2024-03-09T00:04:17.959940+0000","last_unstale":"2024-03-09T00:06:06.271674+0000","last_undegraded":"2024-03-09T00:06:06.271674+0000","last_fullsized":"2024-03-09T00:06:06.271674+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:15:02.999411+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,3,9,5],"acting":[4,3,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1d","version":"84'2","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082791+0000","last_change":"2024-03-09T00:05:05.142560+0000","last_active":"2024-03-09T00:06:06.082791+0000","last_peered":"2024-03-09T00:06:06.082791+0000","last_clean":"2024-03-09T00:06:06.082791+0000","last_became_active":"2024-03-09T00:05:05.139877+0000","last_became_peered":"2024-03-09T00:05:05.139877+0000","last_unstale":"2024-03-09T00:06:06.082791+0000","last_undegraded":"2024-03-09T00:06:06.082791+0000","last_fullsized":"2024-03-09T00:06:06.082791+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:27:30.441723+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,4,7],"acting":[8,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513607+0000","last_change":"2024-03-09T00:04:13.928840+0000","last_active":"2024-03-09T00:06:06.513607+0000","last_peered":"2024-03-09T00:06:06.513607+0000","last_clean":"2024-03-09T00:06:06.513607+0000","last_became_active":"2024-03-09T00:04:13.928442+0000","last_became_peered":"2024-03-09T00:04:13.928442+0000","last_unstale":"2024-03-09T00:06:06.513607+0000","last_undegraded":"2024-03-09T00:06:06.513607+0000","last_fullsized":"2024-03-09T00:06:06.513607+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:41:26.475592+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1a","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271559+0000","last_change":"2024-03-09T00:04:17.955528+0000","last_active":"2024-03-09T00:06:06.271559+0000","last_peered":"2024-03-09T00:06:06.271559+0000","last_clean":"2024-03-09T00:06:06.271559+0000","last_became_active":"2024-03-09T00:04:17.954553+0000","last_became_peered":"2024-03-09T00:04:17.954553+0000","last_unstale":"2024-03-09T00:06:06.271559+0000","last_undegraded":"2024-03-09T00:06:06.271559+0000","last_fullsized":"2024-03-09T00:06:06.271559+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:13:11.767776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,6,5,3],"acting":[4,6,5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.1c","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082413+0000","last_change":"2024-03-09T00:04:13.330016+0000","last_active":"2024-03-09T00:06:06.082413+0000","last_peered":"2024-03-09T00:06:06.082413+0000","last_clean":"2024-03-09T00:06:06.082413+0000","last_became_active":"2024-03-09T00:04:13.329281+0000","last_became_peered":"2024-03-09T00:04:13.329281+0000","last_unstale":"2024-03-09T00:06:06.082413+0000","last_undegraded":"2024-03-09T00:06:06.082413+0000","last_fullsized":"2024-03-09T00:06:06.082413+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:08:36.407025+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"91'15","reported_seq":43,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076341+0000","last_change":"2024-03-09T00:04:13.935147+0000","last_active":"2024-03-09T00:06:06.076341+0000","last_peered":"2024-03-09T00:06:06.076341+0000","last_clean":"2024-03-09T00:06:06.076341+0000","last_became_active":"2024-03-09T00:04:13.934839+0000","last_became_peered":"2024-03-09T00:04:13.934839+0000","last_unstale":"2024-03-09T00:06:06.076341+0000","last_undegraded":"2024-03-09T00:06:06.076341+0000","last_fullsized":"2024-03-09T00:06:06.076341+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:22:37.495629+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1d","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081523+0000","last_change":"2024-03-09T00:04:17.955275+0000","last_active":"2024-03-09T00:06:06.081523+0000","last_peered":"2024-03-09T00:06:06.081523+0000","last_clean":"2024-03-09T00:06:06.081523+0000","last_became_active":"2024-03-09T00:04:17.954853+0000","last_became_peered":"2024-03-09T00:04:17.954853+0000","last_unstale":"2024-03-09T00:06:06.081523+0000","last_undegraded":"2024-03-09T00:06:06.081523+0000","last_fullsized":"2024-03-09T00:06:06.081523+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:42:08.078747+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,7,11],"acting":[2,5,7,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1b","version":"84'1","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075947+0000","last_change":"2024-03-09T00:04:13.331928+0000","last_active":"2024-03-09T00:06:06.075947+0000","last_peered":"2024-03-09T00:06:06.075947+0000","last_clean":"2024-03-09T00:06:06.075947+0000","last_became_active":"2024-03-09T00:04:13.331592+0000","last_became_peered":"2024-03-09T00:04:13.331592+0000","last_unstale":"2024-03-09T00:06:06.075947+0000","last_undegraded":"2024-03-09T00:06:06.075947+0000","last_fullsized":"2024-03-09T00:06:06.075947+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:06:29.511741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082037+0000","last_change":"2024-03-09T00:04:13.931807+0000","last_active":"2024-03-09T00:06:06.082037+0000","last_peered":"2024-03-09T00:06:06.082037+0000","last_clean":"2024-03-09T00:06:06.082037+0000","last_became_active":"2024-03-09T00:04:13.931462+0000","last_became_peered":"2024-03-09T00:04:13.931462+0000","last_unstale":"2024-03-09T00:06:06.082037+0000","last_undegraded":"2024-03-09T00:06:06.082037+0000","last_fullsized":"2024-03-09T00:06:06.082037+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:42:34.627774+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.1c","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081531+0000","last_change":"2024-03-09T00:04:17.955060+0000","last_active":"2024-03-09T00:06:06.081531+0000","last_peered":"2024-03-09T00:06:06.081531+0000","last_clean":"2024-03-09T00:06:06.081531+0000","last_became_active":"2024-03-09T00:04:17.954607+0000","last_became_peered":"2024-03-09T00:04:17.954607+0000","last_unstale":"2024-03-09T00:06:06.081531+0000","last_undegraded":"2024-03-09T00:06:06.081531+0000","last_fullsized":"2024-03-09T00:06:06.081531+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:33:13.348225+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,6,9,5],"acting":[2,6,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1a","version":"84'1","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075185+0000","last_change":"2024-03-09T00:04:13.333721+0000","last_active":"2024-03-09T00:06:06.075185+0000","last_peered":"2024-03-09T00:06:06.075185+0000","last_clean":"2024-03-09T00:06:06.075185+0000","last_became_active":"2024-03-09T00:04:13.333380+0000","last_became_peered":"2024-03-09T00:04:13.333380+0000","last_unstale":"2024-03-09T00:06:06.075185+0000","last_undegraded":"2024-03-09T00:06:06.075185+0000","last_fullsized":"2024-03-09T00:06:06.075185+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:58:55.856485+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"91'8","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076123+0000","last_change":"2024-03-09T00:05:05.056415+0000","last_active":"2024-03-09T00:06:06.076123+0000","last_peered":"2024-03-09T00:06:06.076123+0000","last_clean":"2024-03-09T00:06:06.076123+0000","last_became_active":"2024-03-09T00:05:05.056025+0000","last_became_peered":"2024-03-09T00:05:05.056025+0000","last_unstale":"2024-03-09T00:06:06.076123+0000","last_undegraded":"2024-03-09T00:06:06.076123+0000","last_fullsized":"2024-03-09T00:06:06.076123+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:11:29.970623+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,6,3],"acting":[5,6,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.1f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079682+0000","last_change":"2024-03-09T00:04:17.963418+0000","last_active":"2024-03-09T00:06:06.079682+0000","last_peered":"2024-03-09T00:06:06.079682+0000","last_clean":"2024-03-09T00:06:06.079682+0000","last_became_active":"2024-03-09T00:04:17.963074+0000","last_became_peered":"2024-03-09T00:04:17.963074+0000","last_unstale":"2024-03-09T00:06:06.079682+0000","last_undegraded":"2024-03-09T00:06:06.079682+0000","last_fullsized":"2024-03-09T00:06:06.079682+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:35:58.760174+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,9,0],"acting":[11,4,9,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.19","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076321+0000","last_change":"2024-03-09T00:04:13.328382+0000","last_active":"2024-03-09T00:06:06.076321+0000","last_peered":"2024-03-09T00:06:06.076321+0000","last_clean":"2024-03-09T00:06:06.076321+0000","last_became_active":"2024-03-09T00:04:13.328083+0000","last_became_peered":"2024-03-09T00:04:13.328083+0000","last_unstale":"2024-03-09T00:06:06.076321+0000","last_undegraded":"2024-03-09T00:06:06.076321+0000","last_fullsized":"2024-03-09T00:06:06.076321+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:11:27.577938+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"91'8","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921566+0000","last_change":"2024-03-09T00:04:13.933826+0000","last_active":"2024-03-09T00:06:06.921566+0000","last_peered":"2024-03-09T00:06:06.921566+0000","last_clean":"2024-03-09T00:06:06.921566+0000","last_became_active":"2024-03-09T00:04:13.933484+0000","last_became_peered":"2024-03-09T00:04:13.933484+0000","last_unstale":"2024-03-09T00:06:06.921566+0000","last_undegraded":"2024-03-09T00:06:06.921566+0000","last_fullsized":"2024-03-09T00:06:06.921566+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:53:20.862286+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081208+0000","last_change":"2024-03-09T00:04:17.951970+0000","last_active":"2024-03-09T00:06:06.081208+0000","last_peered":"2024-03-09T00:06:06.081208+0000","last_clean":"2024-03-09T00:06:06.081208+0000","last_became_active":"2024-03-09T00:04:17.951614+0000","last_became_peered":"2024-03-09T00:04:17.951614+0000","last_unstale":"2024-03-09T00:06:06.081208+0000","last_undegraded":"2024-03-09T00:06:06.081208+0000","last_fullsized":"2024-03-09T00:06:06.081208+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:20.240745+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,6,1],"acting":[0,7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.18","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075906+0000","last_change":"2024-03-09T00:04:13.329870+0000","last_active":"2024-03-09T00:06:06.075906+0000","last_peered":"2024-03-09T00:06:06.075906+0000","last_clean":"2024-03-09T00:06:06.075906+0000","last_became_active":"2024-03-09T00:04:13.329564+0000","last_became_peered":"2024-03-09T00:04:13.329564+0000","last_unstale":"2024-03-09T00:06:06.075906+0000","last_undegraded":"2024-03-09T00:06:06.075906+0000","last_fullsized":"2024-03-09T00:06:06.075906+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:43.412510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276794+0000","last_change":"2024-03-09T00:04:13.925730+0000","last_active":"2024-03-09T00:06:06.276794+0000","last_peered":"2024-03-09T00:06:06.276794+0000","last_clean":"2024-03-09T00:06:06.276794+0000","last_became_active":"2024-03-09T00:04:13.925404+0000","last_became_peered":"2024-03-09T00:04:13.925404+0000","last_unstale":"2024-03-09T00:06:06.276794+0000","last_undegraded":"2024-03-09T00:06:06.276794+0000","last_fullsized":"2024-03-09T00:06:06.276794+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:47:35.383805+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"4.11","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921459+0000","last_change":"2024-03-09T00:04:17.955093+0000","last_active":"2024-03-09T00:06:06.921459+0000","last_peered":"2024-03-09T00:06:06.921459+0000","last_clean":"2024-03-09T00:06:06.921459+0000","last_became_active":"2024-03-09T00:04:17.954620+0000","last_became_peered":"2024-03-09T00:04:17.954620+0000","last_unstale":"2024-03-09T00:06:06.921459+0000","last_undegraded":"2024-03-09T00:06:06.921459+0000","last_fullsized":"2024-03-09T00:06:06.921459+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:12:37.343246+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7,1,5],"acting":[3,7,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.17","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075432+0000","last_change":"2024-03-09T00:04:13.334614+0000","last_active":"2024-03-09T00:06:06.075432+0000","last_peered":"2024-03-09T00:06:06.075432+0000","last_clean":"2024-03-09T00:06:06.075432+0000","last_became_active":"2024-03-09T00:04:13.334297+0000","last_became_peered":"2024-03-09T00:04:13.334297+0000","last_unstale":"2024-03-09T00:06:06.075432+0000","last_undegraded":"2024-03-09T00:06:06.075432+0000","last_fullsized":"2024-03-09T00:06:06.075432+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:13:40.940956+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"91'10","reported_seq":39,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513829+0000","last_change":"2024-03-09T00:04:13.928240+0000","last_active":"2024-03-09T00:06:06.513829+0000","last_peered":"2024-03-09T00:06:06.513829+0000","last_clean":"2024-03-09T00:06:06.513829+0000","last_became_active":"2024-03-09T00:04:13.927767+0000","last_became_peered":"2024-03-09T00:04:13.927767+0000","last_unstale":"2024-03-09T00:06:06.513829+0000","last_undegraded":"2024-03-09T00:06:06.513829+0000","last_fullsized":"2024-03-09T00:06:06.513829+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:10:15.694185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.10","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081694+0000","last_change":"2024-03-09T00:05:05.139426+0000","last_active":"2024-03-09T00:06:06.081694+0000","last_peered":"2024-03-09T00:06:06.081694+0000","last_clean":"2024-03-09T00:06:06.081694+0000","last_became_active":"2024-03-09T00:05:05.138971+0000","last_became_peered":"2024-03-09T00:05:05.138971+0000","last_unstale":"2024-03-09T00:06:06.081694+0000","last_undegraded":"2024-03-09T00:06:06.081694+0000","last_fullsized":"2024-03-09T00:06:06.081694+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:40:59.870972+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2,4],"acting":[8,0,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.16","version":"0'0","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.514952+0000","last_change":"2024-03-09T00:06:06.514952+0000","last_active":"2024-03-09T00:06:06.514952+0000","last_peered":"2024-03-09T00:06:06.514952+0000","last_clean":"2024-03-09T00:06:06.514952+0000","last_became_active":"2024-03-09T00:06:06.514480+0000","last_became_peered":"2024-03-09T00:06:06.514480+0000","last_unstale":"2024-03-09T00:06:06.514952+0000","last_undegraded":"2024-03-09T00:06:06.514952+0000","last_fullsized":"2024-03-09T00:06:06.514952+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:03.976364+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,4],"acting":[7,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081301+0000","last_change":"2024-03-09T00:04:13.936257+0000","last_active":"2024-03-09T00:06:06.081301+0000","last_peered":"2024-03-09T00:06:06.081301+0000","last_clean":"2024-03-09T00:06:06.081301+0000","last_became_active":"2024-03-09T00:04:13.935439+0000","last_became_peered":"2024-03-09T00:04:13.935439+0000","last_unstale":"2024-03-09T00:06:06.081301+0000","last_undegraded":"2024-03-09T00:06:06.081301+0000","last_fullsized":"2024-03-09T00:06:06.081301+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:28:50.884796+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.13","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271940+0000","last_change":"2024-03-09T00:04:17.957580+0000","last_active":"2024-03-09T00:06:06.271940+0000","last_peered":"2024-03-09T00:06:06.271940+0000","last_clean":"2024-03-09T00:06:06.271940+0000","last_became_active":"2024-03-09T00:04:17.957161+0000","last_became_peered":"2024-03-09T00:04:17.957161+0000","last_unstale":"2024-03-09T00:06:06.271940+0000","last_undegraded":"2024-03-09T00:06:06.271940+0000","last_fullsized":"2024-03-09T00:06:06.271940+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:41:18.137931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,1,2,9],"acting":[4,1,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.15","version":"84'3","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076324+0000","last_change":"2024-03-09T00:04:13.331832+0000","last_active":"2024-03-09T00:06:06.076324+0000","last_peered":"2024-03-09T00:06:06.076324+0000","last_clean":"2024-03-09T00:06:06.076324+0000","last_became_active":"2024-03-09T00:04:13.329440+0000","last_became_peered":"2024-03-09T00:04:13.329440+0000","last_unstale":"2024-03-09T00:06:06.076324+0000","last_undegraded":"2024-03-09T00:06:06.076324+0000","last_fullsized":"2024-03-09T00:06:06.076324+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:39:55.743581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"91'11","reported_seq":41,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513828+0000","last_change":"2024-03-09T00:04:13.935444+0000","last_active":"2024-03-09T00:06:06.513828+0000","last_peered":"2024-03-09T00:06:06.513828+0000","last_clean":"2024-03-09T00:06:06.513828+0000","last_became_active":"2024-03-09T00:04:13.935113+0000","last_became_peered":"2024-03-09T00:04:13.935113+0000","last_unstale":"2024-03-09T00:06:06.513828+0000","last_undegraded":"2024-03-09T00:06:06.513828+0000","last_fullsized":"2024-03-09T00:06:06.513828+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:22:34.928646+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.12","version":"0'0","reported_seq":16,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.074900+0000","last_change":"2024-03-09T00:05:05.058517+0000","last_active":"2024-03-09T00:06:06.074900+0000","last_peered":"2024-03-09T00:06:06.074900+0000","last_clean":"2024-03-09T00:06:06.074900+0000","last_became_active":"2024-03-09T00:05:05.057877+0000","last_became_peered":"2024-03-09T00:05:05.057877+0000","last_unstale":"2024-03-09T00:06:06.074900+0000","last_undegraded":"2024-03-09T00:06:06.074900+0000","last_fullsized":"2024-03-09T00:06:06.074900+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:42:49.424790+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,11,5],"acting":[10,8,11,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.14","version":"93'36","reported_seq":320,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:39.198178+0000","last_change":"2024-03-09T00:04:13.329688+0000","last_active":"2024-03-09T00:06:39.198178+0000","last_peered":"2024-03-09T00:06:39.198178+0000","last_clean":"2024-03-09T00:06:39.198178+0000","last_became_active":"2024-03-09T00:04:13.329374+0000","last_became_peered":"2024-03-09T00:04:13.329374+0000","last_unstale":"2024-03-09T00:06:39.198178+0000","last_undegraded":"2024-03-09T00:06:39.198178+0000","last_fullsized":"2024-03-09T00:06:39.198178+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":36,"log_dups_size":0,"ondisk_log_size":36,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:54.571216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":69286,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":252,"num_read_kb":194,"num_write":36,"num_write_kb":87,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082439+0000","last_change":"2024-03-09T00:04:13.930282+0000","last_active":"2024-03-09T00:06:06.082439+0000","last_peered":"2024-03-09T00:06:06.082439+0000","last_clean":"2024-03-09T00:06:06.082439+0000","last_became_active":"2024-03-09T00:04:13.929947+0000","last_became_peered":"2024-03-09T00:04:13.929947+0000","last_unstale":"2024-03-09T00:06:06.082439+0000","last_undegraded":"2024-03-09T00:06:06.082439+0000","last_fullsized":"2024-03-09T00:06:06.082439+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:22.443786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.15","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076594+0000","last_change":"2024-03-09T00:04:17.955585+0000","last_active":"2024-03-09T00:06:06.076594+0000","last_peered":"2024-03-09T00:06:06.076594+0000","last_clean":"2024-03-09T00:06:06.076594+0000","last_became_active":"2024-03-09T00:04:17.954132+0000","last_became_peered":"2024-03-09T00:04:17.954132+0000","last_unstale":"2024-03-09T00:06:06.076594+0000","last_undegraded":"2024-03-09T00:06:06.076594+0000","last_fullsized":"2024-03-09T00:06:06.076594+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:17:45.184081+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,8,3,6],"acting":[5,8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.13","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081920+0000","last_change":"2024-03-09T00:04:13.328131+0000","last_active":"2024-03-09T00:06:06.081920+0000","last_peered":"2024-03-09T00:06:06.081920+0000","last_clean":"2024-03-09T00:06:06.081920+0000","last_became_active":"2024-03-09T00:04:13.327792+0000","last_became_peered":"2024-03-09T00:04:13.327792+0000","last_unstale":"2024-03-09T00:06:06.081920+0000","last_undegraded":"2024-03-09T00:06:06.081920+0000","last_fullsized":"2024-03-09T00:06:06.081920+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:08:43.307532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":34,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":3,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076436+0000","last_change":"2024-03-09T00:04:13.939201+0000","last_active":"2024-03-09T00:06:06.076436+0000","last_peered":"2024-03-09T00:06:06.076436+0000","last_clean":"2024-03-09T00:06:06.076436+0000","last_became_active":"2024-03-09T00:04:13.938846+0000","last_became_peered":"2024-03-09T00:04:13.938846+0000","last_unstale":"2024-03-09T00:06:06.076436+0000","last_undegraded":"2024-03-09T00:06:06.076436+0000","last_fullsized":"2024-03-09T00:06:06.076436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:06:44.624504+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.14","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079285+0000","last_change":"2024-03-09T00:04:17.960815+0000","last_active":"2024-03-09T00:06:06.079285+0000","last_peered":"2024-03-09T00:06:06.079285+0000","last_clean":"2024-03-09T00:06:06.079285+0000","last_became_active":"2024-03-09T00:04:17.960389+0000","last_became_peered":"2024-03-09T00:04:17.960389+0000","last_unstale":"2024-03-09T00:06:06.079285+0000","last_undegraded":"2024-03-09T00:06:06.079285+0000","last_fullsized":"2024-03-09T00:06:06.079285+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:58:44.777866+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,7,6],"acting":[11,9,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.12","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075921+0000","last_change":"2024-03-09T00:04:13.329350+0000","last_active":"2024-03-09T00:06:06.075921+0000","last_peered":"2024-03-09T00:06:06.075921+0000","last_clean":"2024-03-09T00:06:06.075921+0000","last_became_active":"2024-03-09T00:04:13.329040+0000","last_became_peered":"2024-03-09T00:04:13.329040+0000","last_unstale":"2024-03-09T00:06:06.075921+0000","last_undegraded":"2024-03-09T00:06:06.075921+0000","last_fullsized":"2024-03-09T00:06:06.075921+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:44:30.128763+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"91'15","reported_seq":50,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079599+0000","last_change":"2024-03-09T00:05:05.059283+0000","last_active":"2024-03-09T00:06:06.079599+0000","last_peered":"2024-03-09T00:06:06.079599+0000","last_clean":"2024-03-09T00:06:06.079599+0000","last_became_active":"2024-03-09T00:05:05.058822+0000","last_became_peered":"2024-03-09T00:05:05.058822+0000","last_unstale":"2024-03-09T00:06:06.079599+0000","last_undegraded":"2024-03-09T00:06:06.079599+0000","last_fullsized":"2024-03-09T00:06:06.079599+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:13:02.253810+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.17","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276881+0000","last_change":"2024-03-09T00:04:17.963178+0000","last_active":"2024-03-09T00:06:06.276881+0000","last_peered":"2024-03-09T00:06:06.276881+0000","last_clean":"2024-03-09T00:06:06.276881+0000","last_became_active":"2024-03-09T00:04:17.962767+0000","last_became_peered":"2024-03-09T00:04:17.962767+0000","last_unstale":"2024-03-09T00:06:06.276881+0000","last_undegraded":"2024-03-09T00:06:06.276881+0000","last_fullsized":"2024-03-09T00:06:06.276881+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:29.282181+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,0,3],"acting":[9,7,0,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.11","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271661+0000","last_change":"2024-03-09T00:04:13.328884+0000","last_active":"2024-03-09T00:06:06.271661+0000","last_peered":"2024-03-09T00:06:06.271661+0000","last_clean":"2024-03-09T00:06:06.271661+0000","last_became_active":"2024-03-09T00:04:13.328539+0000","last_became_peered":"2024-03-09T00:04:13.328539+0000","last_unstale":"2024-03-09T00:06:06.271661+0000","last_undegraded":"2024-03-09T00:06:06.271661+0000","last_fullsized":"2024-03-09T00:06:06.271661+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:10:45.532407+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079467+0000","last_change":"2024-03-09T00:04:13.932191+0000","last_active":"2024-03-09T00:06:06.079467+0000","last_peered":"2024-03-09T00:06:06.079467+0000","last_clean":"2024-03-09T00:06:06.079467+0000","last_became_active":"2024-03-09T00:04:13.931890+0000","last_became_peered":"2024-03-09T00:04:13.931890+0000","last_unstale":"2024-03-09T00:06:06.079467+0000","last_undegraded":"2024-03-09T00:06:06.079467+0000","last_fullsized":"2024-03-09T00:06:06.079467+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:52:33.906764+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.16","version":"0'0","reported_seq":22,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076789+0000","last_change":"2024-03-09T00:04:17.954650+0000","last_active":"2024-03-09T00:06:06.076789+0000","last_peered":"2024-03-09T00:06:06.076789+0000","last_clean":"2024-03-09T00:06:06.076789+0000","last_became_active":"2024-03-09T00:04:17.954246+0000","last_became_peered":"2024-03-09T00:04:17.954246+0000","last_unstale":"2024-03-09T00:06:06.076789+0000","last_undegraded":"2024-03-09T00:06:06.076789+0000","last_fullsized":"2024-03-09T00:06:06.076789+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:30:15.141831+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,9,11],"acting":[6,3,9,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.10","version":"84'8","reported_seq":21,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.089907+0000","last_change":"2024-03-09T00:06:06.089420+0000","last_active":"2024-03-09T00:06:06.089907+0000","last_peered":"2024-03-09T00:06:06.089907+0000","last_clean":"2024-03-09T00:06:06.089907+0000","last_became_active":"2024-03-09T00:06:06.083164+0000","last_became_peered":"2024-03-09T00:06:06.083164+0000","last_unstale":"2024-03-09T00:06:06.089907+0000","last_undegraded":"2024-03-09T00:06:06.089907+0000","last_fullsized":"2024-03-09T00:06:06.089907+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:08:34.770550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.11","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075880+0000","last_change":"2024-03-09T00:04:13.932133+0000","last_active":"2024-03-09T00:06:06.075880+0000","last_peered":"2024-03-09T00:06:06.075880+0000","last_clean":"2024-03-09T00:06:06.075880+0000","last_became_active":"2024-03-09T00:04:13.931786+0000","last_became_peered":"2024-03-09T00:04:13.931786+0000","last_unstale":"2024-03-09T00:06:06.075880+0000","last_undegraded":"2024-03-09T00:06:06.075880+0000","last_fullsized":"2024-03-09T00:06:06.075880+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:11:11.692204+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"4.9","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075786+0000","last_change":"2024-03-09T00:04:17.956967+0000","last_active":"2024-03-09T00:06:06.075786+0000","last_peered":"2024-03-09T00:06:06.075786+0000","last_clean":"2024-03-09T00:06:06.075786+0000","last_became_active":"2024-03-09T00:04:17.956524+0000","last_became_peered":"2024-03-09T00:04:17.956524+0000","last_unstale":"2024-03-09T00:06:06.075786+0000","last_undegraded":"2024-03-09T00:06:06.075786+0000","last_fullsized":"2024-03-09T00:06:06.075786+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:13:40.437581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,3,1],"acting":[5,0,3,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"84'1","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082961+0000","last_change":"2024-03-09T00:04:13.330589+0000","last_active":"2024-03-09T00:06:06.082961+0000","last_peered":"2024-03-09T00:06:06.082961+0000","last_clean":"2024-03-09T00:06:06.082961+0000","last_became_active":"2024-03-09T00:04:13.329826+0000","last_became_peered":"2024-03-09T00:04:13.329826+0000","last_unstale":"2024-03-09T00:06:06.082961+0000","last_undegraded":"2024-03-09T00:06:06.082961+0000","last_fullsized":"2024-03-09T00:06:06.082961+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:06:20.497392+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"91'5","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079668+0000","last_change":"2024-03-09T00:05:05.138598+0000","last_active":"2024-03-09T00:06:06.079668+0000","last_peered":"2024-03-09T00:06:06.079668+0000","last_clean":"2024-03-09T00:06:06.079668+0000","last_became_active":"2024-03-09T00:05:05.138166+0000","last_became_peered":"2024-03-09T00:05:05.138166+0000","last_unstale":"2024-03-09T00:06:06.079668+0000","last_undegraded":"2024-03-09T00:06:06.079668+0000","last_fullsized":"2024-03-09T00:06:06.079668+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:59:42.864722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,9,4],"acting":[11,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082189+0000","last_change":"2024-03-09T00:04:18.031536+0000","last_active":"2024-03-09T00:06:06.082189+0000","last_peered":"2024-03-09T00:06:06.082189+0000","last_clean":"2024-03-09T00:06:06.082189+0000","last_became_active":"2024-03-09T00:04:18.031031+0000","last_became_peered":"2024-03-09T00:04:18.031031+0000","last_unstale":"2024-03-09T00:06:06.082189+0000","last_undegraded":"2024-03-09T00:06:06.082189+0000","last_fullsized":"2024-03-09T00:06:06.082189+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:44:28.393929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,10,11,9],"acting":[8,10,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276948+0000","last_change":"2024-03-09T00:04:13.335842+0000","last_active":"2024-03-09T00:06:06.276948+0000","last_peered":"2024-03-09T00:06:06.276948+0000","last_clean":"2024-03-09T00:06:06.276948+0000","last_became_active":"2024-03-09T00:04:13.335517+0000","last_became_peered":"2024-03-09T00:04:13.335517+0000","last_unstale":"2024-03-09T00:06:06.276948+0000","last_undegraded":"2024-03-09T00:06:06.276948+0000","last_fullsized":"2024-03-09T00:06:06.276948+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:18:02.508851+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"91'4","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075133+0000","last_change":"2024-03-09T00:04:13.933044+0000","last_active":"2024-03-09T00:06:06.075133+0000","last_peered":"2024-03-09T00:06:06.075133+0000","last_clean":"2024-03-09T00:06:06.075133+0000","last_became_active":"2024-03-09T00:04:13.932700+0000","last_became_peered":"2024-03-09T00:04:13.932700+0000","last_unstale":"2024-03-09T00:06:06.075133+0000","last_undegraded":"2024-03-09T00:06:06.075133+0000","last_fullsized":"2024-03-09T00:06:06.075133+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:24:13.124122+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076732+0000","last_change":"2024-03-09T00:04:17.949016+0000","last_active":"2024-03-09T00:06:06.076732+0000","last_peered":"2024-03-09T00:06:06.076732+0000","last_clean":"2024-03-09T00:06:06.076732+0000","last_became_active":"2024-03-09T00:04:17.948625+0000","last_became_peered":"2024-03-09T00:04:17.948625+0000","last_unstale":"2024-03-09T00:06:06.076732+0000","last_undegraded":"2024-03-09T00:06:06.076732+0000","last_fullsized":"2024-03-09T00:06:06.076732+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:15:10.265821+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,0,1,8],"acting":[6,0,1,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079746+0000","last_change":"2024-03-09T00:04:13.334550+0000","last_active":"2024-03-09T00:06:06.079746+0000","last_peered":"2024-03-09T00:06:06.079746+0000","last_clean":"2024-03-09T00:06:06.079746+0000","last_became_active":"2024-03-09T00:04:13.334202+0000","last_became_peered":"2024-03-09T00:04:13.334202+0000","last_unstale":"2024-03-09T00:06:06.079746+0000","last_undegraded":"2024-03-09T00:06:06.079746+0000","last_fullsized":"2024-03-09T00:06:06.079746+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:05:39.185051+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"91'7","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075373+0000","last_change":"2024-03-09T00:04:13.931924+0000","last_active":"2024-03-09T00:06:06.075373+0000","last_peered":"2024-03-09T00:06:06.075373+0000","last_clean":"2024-03-09T00:06:06.075373+0000","last_became_active":"2024-03-09T00:04:13.931327+0000","last_became_peered":"2024-03-09T00:04:13.931327+0000","last_unstale":"2024-03-09T00:06:06.075373+0000","last_undegraded":"2024-03-09T00:06:06.075373+0000","last_fullsized":"2024-03-09T00:06:06.075373+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:21:43.012507+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":30,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076899+0000","last_change":"2024-03-09T00:05:05.138164+0000","last_active":"2024-03-09T00:06:06.076899+0000","last_peered":"2024-03-09T00:06:06.076899+0000","last_clean":"2024-03-09T00:06:06.076899+0000","last_became_active":"2024-03-09T00:05:05.137633+0000","last_became_peered":"2024-03-09T00:05:05.137633+0000","last_unstale":"2024-03-09T00:06:06.076899+0000","last_undegraded":"2024-03-09T00:06:06.076899+0000","last_fullsized":"2024-03-09T00:06:06.076899+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:12:44.310276+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,8,10,0],"acting":[6,8,10,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.0","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076054+0000","last_change":"2024-03-09T00:04:13.933896+0000","last_active":"2024-03-09T00:06:06.076054+0000","last_peered":"2024-03-09T00:06:06.076054+0000","last_clean":"2024-03-09T00:06:06.076054+0000","last_became_active":"2024-03-09T00:04:13.932039+0000","last_became_peered":"2024-03-09T00:04:13.932039+0000","last_unstale":"2024-03-09T00:06:06.076054+0000","last_undegraded":"2024-03-09T00:06:06.076054+0000","last_fullsized":"2024-03-09T00:06:06.076054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:10:22.331776+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076336+0000","last_change":"2024-03-09T00:04:13.333972+0000","last_active":"2024-03-09T00:06:06.076336+0000","last_peered":"2024-03-09T00:06:06.076336+0000","last_clean":"2024-03-09T00:06:06.076336+0000","last_became_active":"2024-03-09T00:04:13.333671+0000","last_became_peered":"2024-03-09T00:04:13.333671+0000","last_unstale":"2024-03-09T00:06:06.076336+0000","last_undegraded":"2024-03-09T00:06:06.076336+0000","last_fullsized":"2024-03-09T00:06:06.076336+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:06:06.473963+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.5","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079810+0000","last_change":"2024-03-09T00:04:17.955169+0000","last_active":"2024-03-09T00:06:06.079810+0000","last_peered":"2024-03-09T00:06:06.079810+0000","last_clean":"2024-03-09T00:06:06.079810+0000","last_became_active":"2024-03-09T00:04:17.954766+0000","last_became_peered":"2024-03-09T00:04:17.954766+0000","last_unstale":"2024-03-09T00:06:06.079810+0000","last_undegraded":"2024-03-09T00:06:06.079810+0000","last_fullsized":"2024-03-09T00:06:06.079810+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:25:27.420156+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,11,9],"acting":[7,5,11,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.2","version":"91'8","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.276945+0000","last_change":"2024-03-09T00:05:05.058253+0000","last_active":"2024-03-09T00:06:06.276945+0000","last_peered":"2024-03-09T00:06:06.276945+0000","last_clean":"2024-03-09T00:06:06.276945+0000","last_became_active":"2024-03-09T00:05:05.057864+0000","last_became_peered":"2024-03-09T00:05:05.057864+0000","last_unstale":"2024-03-09T00:06:06.276945+0000","last_undegraded":"2024-03-09T00:06:06.276945+0000","last_fullsized":"2024-03-09T00:06:06.276945+0000","mapping_epoch":90,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":91,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:33:14.993355+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,1],"acting":[9,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271946+0000","last_change":"2024-03-09T00:04:13.329204+0000","last_active":"2024-03-09T00:06:06.271946+0000","last_peered":"2024-03-09T00:06:06.271946+0000","last_clean":"2024-03-09T00:06:06.271946+0000","last_became_active":"2024-03-09T00:04:13.328377+0000","last_became_peered":"2024-03-09T00:04:13.328377+0000","last_unstale":"2024-03-09T00:06:06.271946+0000","last_undegraded":"2024-03-09T00:06:06.271946+0000","last_fullsized":"2024-03-09T00:06:06.271946+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:39:46.322399+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"22'137","reported_seq":364,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079321+0000","last_change":"2024-03-09T00:02:57.239137+0000","last_active":"2024-03-09T00:06:06.079321+0000","last_peered":"2024-03-09T00:06:06.079321+0000","last_clean":"2024-03-09T00:06:06.079321+0000","last_became_active":"2024-03-09T00:02:57.238532+0000","last_became_peered":"2024-03-09T00:02:57.238532+0000","last_unstale":"2024-03-09T00:06:06.079321+0000","last_undegraded":"2024-03-09T00:06:06.079321+0000","last_fullsized":"2024-03-09T00:06:06.079321+0000","mapping_epoch":75,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":76,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-08T23:59:30.529805+0000","last_clean_scrub_stamp":"2024-03-08T23:59:30.529805+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:51:01.157455+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076429+0000","last_change":"2024-03-09T00:04:18.032177+0000","last_active":"2024-03-09T00:06:06.076429+0000","last_peered":"2024-03-09T00:06:06.076429+0000","last_clean":"2024-03-09T00:06:06.076429+0000","last_became_active":"2024-03-09T00:04:18.031705+0000","last_became_peered":"2024-03-09T00:04:18.031705+0000","last_unstale":"2024-03-09T00:06:06.076429+0000","last_undegraded":"2024-03-09T00:06:06.076429+0000","last_fullsized":"2024-03-09T00:06:06.076429+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:04:26.071589+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,6,10],"acting":[5,9,6,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076267+0000","last_change":"2024-03-09T00:04:13.930174+0000","last_active":"2024-03-09T00:06:06.076267+0000","last_peered":"2024-03-09T00:06:06.076267+0000","last_clean":"2024-03-09T00:06:06.076267+0000","last_became_active":"2024-03-09T00:04:13.929850+0000","last_became_peered":"2024-03-09T00:04:13.929850+0000","last_unstale":"2024-03-09T00:06:06.076267+0000","last_undegraded":"2024-03-09T00:06:06.076267+0000","last_fullsized":"2024-03-09T00:06:06.076267+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:24:14.774924+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079600+0000","last_change":"2024-03-09T00:04:13.336005+0000","last_active":"2024-03-09T00:06:06.079600+0000","last_peered":"2024-03-09T00:06:06.079600+0000","last_clean":"2024-03-09T00:06:06.079600+0000","last_became_active":"2024-03-09T00:04:13.335712+0000","last_became_peered":"2024-03-09T00:04:13.335712+0000","last_unstale":"2024-03-09T00:06:06.079600+0000","last_undegraded":"2024-03-09T00:06:06.079600+0000","last_fullsized":"2024-03-09T00:06:06.079600+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:49:31.719020+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079335+0000","last_change":"2024-03-09T00:04:17.959331+0000","last_active":"2024-03-09T00:06:06.079335+0000","last_peered":"2024-03-09T00:06:06.079335+0000","last_clean":"2024-03-09T00:06:06.079335+0000","last_became_active":"2024-03-09T00:04:17.958925+0000","last_became_peered":"2024-03-09T00:04:17.958925+0000","last_unstale":"2024-03-09T00:06:06.079335+0000","last_undegraded":"2024-03-09T00:06:06.079335+0000","last_fullsized":"2024-03-09T00:06:06.079335+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:40:15.673888+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,7,6],"acting":[11,5,7,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":11,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.278451+0000","last_change":"2024-03-09T00:06:06.278068+0000","last_active":"2024-03-09T00:06:06.278451+0000","last_peered":"2024-03-09T00:06:06.278451+0000","last_clean":"2024-03-09T00:06:06.278451+0000","last_became_active":"2024-03-09T00:06:06.277772+0000","last_became_peered":"2024-03-09T00:06:06.277772+0000","last_unstale":"2024-03-09T00:06:06.278451+0000","last_undegraded":"2024-03-09T00:06:06.278451+0000","last_fullsized":"2024-03-09T00:06:06.278451+0000","mapping_epoch":92,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":93,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:36:33.637238+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,9],"acting":[11,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.4","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076841+0000","last_change":"2024-03-09T00:04:13.932389+0000","last_active":"2024-03-09T00:06:06.076841+0000","last_peered":"2024-03-09T00:06:06.076841+0000","last_clean":"2024-03-09T00:06:06.076841+0000","last_became_active":"2024-03-09T00:04:13.930878+0000","last_became_peered":"2024-03-09T00:04:13.930878+0000","last_unstale":"2024-03-09T00:06:06.076841+0000","last_undegraded":"2024-03-09T00:06:06.076841+0000","last_fullsized":"2024-03-09T00:06:06.076841+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:49:29.609904+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921495+0000","last_change":"2024-03-09T00:04:18.031294+0000","last_active":"2024-03-09T00:06:06.921495+0000","last_peered":"2024-03-09T00:06:06.921495+0000","last_clean":"2024-03-09T00:06:06.921495+0000","last_became_active":"2024-03-09T00:04:18.030893+0000","last_became_peered":"2024-03-09T00:04:18.030893+0000","last_unstale":"2024-03-09T00:06:06.921495+0000","last_undegraded":"2024-03-09T00:06:06.921495+0000","last_fullsized":"2024-03-09T00:06:06.921495+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:47.510847+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,10,0,7],"acting":[3,10,0,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"93'8","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:23.979195+0000","last_change":"2024-03-09T00:04:13.341106+0000","last_active":"2024-03-09T00:06:23.979195+0000","last_peered":"2024-03-09T00:06:23.979195+0000","last_clean":"2024-03-09T00:06:23.979195+0000","last_became_active":"2024-03-09T00:04:13.340781+0000","last_became_peered":"2024-03-09T00:04:13.340781+0000","last_unstale":"2024-03-09T00:06:23.979195+0000","last_undegraded":"2024-03-09T00:06:23.979195+0000","last_fullsized":"2024-03-09T00:06:23.979195+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:28:40.391786+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":15,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921938+0000","last_change":"2024-03-09T00:04:13.933588+0000","last_active":"2024-03-09T00:06:06.921938+0000","last_peered":"2024-03-09T00:06:06.921938+0000","last_clean":"2024-03-09T00:06:06.921938+0000","last_became_active":"2024-03-09T00:04:13.933218+0000","last_became_peered":"2024-03-09T00:04:13.933218+0000","last_unstale":"2024-03-09T00:06:06.921938+0000","last_undegraded":"2024-03-09T00:06:06.921938+0000","last_fullsized":"2024-03-09T00:06:06.921938+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:21:33.847909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081216+0000","last_change":"2024-03-09T00:04:17.951073+0000","last_active":"2024-03-09T00:06:06.081216+0000","last_peered":"2024-03-09T00:06:06.081216+0000","last_clean":"2024-03-09T00:06:06.081216+0000","last_became_active":"2024-03-09T00:04:17.950648+0000","last_became_peered":"2024-03-09T00:04:17.950648+0000","last_unstale":"2024-03-09T00:06:06.081216+0000","last_undegraded":"2024-03-09T00:06:06.081216+0000","last_fullsized":"2024-03-09T00:06:06.081216+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:07.038146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1,4,7],"acting":[2,1,4,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.7","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.272193+0000","last_change":"2024-03-09T00:04:13.334984+0000","last_active":"2024-03-09T00:06:06.272193+0000","last_peered":"2024-03-09T00:06:06.272193+0000","last_clean":"2024-03-09T00:06:06.272193+0000","last_became_active":"2024-03-09T00:04:13.334631+0000","last_became_peered":"2024-03-09T00:04:13.334631+0000","last_unstale":"2024-03-09T00:06:06.272193+0000","last_undegraded":"2024-03-09T00:06:06.272193+0000","last_fullsized":"2024-03-09T00:06:06.272193+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:31:16.593480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081386+0000","last_change":"2024-03-09T00:04:13.934941+0000","last_active":"2024-03-09T00:06:06.081386+0000","last_peered":"2024-03-09T00:06:06.081386+0000","last_clean":"2024-03-09T00:06:06.081386+0000","last_became_active":"2024-03-09T00:04:13.934616+0000","last_became_peered":"2024-03-09T00:04:13.934616+0000","last_unstale":"2024-03-09T00:06:06.081386+0000","last_undegraded":"2024-03-09T00:06:06.081386+0000","last_fullsized":"2024-03-09T00:06:06.081386+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:18:58.475892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.e","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271932+0000","last_change":"2024-03-09T00:04:18.032519+0000","last_active":"2024-03-09T00:06:06.271932+0000","last_peered":"2024-03-09T00:06:06.271932+0000","last_clean":"2024-03-09T00:06:06.271932+0000","last_became_active":"2024-03-09T00:04:18.031819+0000","last_became_peered":"2024-03-09T00:04:18.031819+0000","last_unstale":"2024-03-09T00:06:06.271932+0000","last_undegraded":"2024-03-09T00:06:06.271932+0000","last_fullsized":"2024-03-09T00:06:06.271932+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:30:03.586993+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5,10,9],"acting":[4,5,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.8","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513561+0000","last_change":"2024-03-09T00:04:13.330503+0000","last_active":"2024-03-09T00:06:06.513561+0000","last_peered":"2024-03-09T00:06:06.513561+0000","last_clean":"2024-03-09T00:06:06.513561+0000","last_became_active":"2024-03-09T00:04:13.330186+0000","last_became_peered":"2024-03-09T00:04:13.330186+0000","last_unstale":"2024-03-09T00:06:06.513561+0000","last_undegraded":"2024-03-09T00:06:06.513561+0000","last_fullsized":"2024-03-09T00:06:06.513561+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:03:19.318984+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"91'2","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081820+0000","last_change":"2024-03-09T00:04:13.927812+0000","last_active":"2024-03-09T00:06:06.081820+0000","last_peered":"2024-03-09T00:06:06.081820+0000","last_clean":"2024-03-09T00:06:06.081820+0000","last_became_active":"2024-03-09T00:04:13.927285+0000","last_became_peered":"2024-03-09T00:04:13.927285+0000","last_unstale":"2024-03-09T00:06:06.081820+0000","last_undegraded":"2024-03-09T00:06:06.081820+0000","last_fullsized":"2024-03-09T00:06:06.081820+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:11:15.720965+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"4.f","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.277174+0000","last_change":"2024-03-09T00:04:17.953066+0000","last_active":"2024-03-09T00:06:06.277174+0000","last_peered":"2024-03-09T00:06:06.277174+0000","last_clean":"2024-03-09T00:06:06.277174+0000","last_became_active":"2024-03-09T00:04:17.952234+0000","last_became_peered":"2024-03-09T00:04:17.952234+0000","last_unstale":"2024-03-09T00:06:06.277174+0000","last_undegraded":"2024-03-09T00:06:06.277174+0000","last_fullsized":"2024-03-09T00:06:06.277174+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:05:41.786484+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,2,5],"acting":[9,3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.9","version":"0'0","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921579+0000","last_change":"2024-03-09T00:04:13.334464+0000","last_active":"2024-03-09T00:06:06.921579+0000","last_peered":"2024-03-09T00:06:06.921579+0000","last_clean":"2024-03-09T00:06:06.921579+0000","last_became_active":"2024-03-09T00:04:13.334117+0000","last_became_peered":"2024-03-09T00:04:13.334117+0000","last_unstale":"2024-03-09T00:06:06.921579+0000","last_undegraded":"2024-03-09T00:06:06.921579+0000","last_fullsized":"2024-03-09T00:06:06.921579+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:15:55.950661+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079350+0000","last_change":"2024-03-09T00:04:13.933139+0000","last_active":"2024-03-09T00:06:06.079350+0000","last_peered":"2024-03-09T00:06:06.079350+0000","last_clean":"2024-03-09T00:06:06.079350+0000","last_became_active":"2024-03-09T00:04:13.932826+0000","last_became_peered":"2024-03-09T00:04:13.932826+0000","last_unstale":"2024-03-09T00:06:06.079350+0000","last_undegraded":"2024-03-09T00:06:06.079350+0000","last_fullsized":"2024-03-09T00:06:06.079350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T06:18:25.479581+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"4.c","version":"89'1","reported_seq":24,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271971+0000","last_change":"2024-03-09T00:04:17.957558+0000","last_active":"2024-03-09T00:06:06.271971+0000","last_peered":"2024-03-09T00:06:06.271971+0000","last_clean":"2024-03-09T00:06:06.271971+0000","last_became_active":"2024-03-09T00:04:17.957085+0000","last_became_peered":"2024-03-09T00:04:17.957085+0000","last_unstale":"2024-03-09T00:06:06.271971+0000","last_undegraded":"2024-03-09T00:06:06.271971+0000","last_fullsized":"2024-03-09T00:06:06.271971+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T01:43:51.874357+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,9,7,8],"acting":[4,9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.a","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081894+0000","last_change":"2024-03-09T00:04:13.325071+0000","last_active":"2024-03-09T00:06:06.081894+0000","last_peered":"2024-03-09T00:06:06.081894+0000","last_clean":"2024-03-09T00:06:06.081894+0000","last_became_active":"2024-03-09T00:04:13.324700+0000","last_became_peered":"2024-03-09T00:04:13.324700+0000","last_unstale":"2024-03-09T00:06:06.081894+0000","last_undegraded":"2024-03-09T00:06:06.081894+0000","last_fullsized":"2024-03-09T00:06:06.081894+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:09:29.308587+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921869+0000","last_change":"2024-03-09T00:04:13.927634+0000","last_active":"2024-03-09T00:06:06.921869+0000","last_peered":"2024-03-09T00:06:06.921869+0000","last_clean":"2024-03-09T00:06:06.921869+0000","last_became_active":"2024-03-09T00:04:13.927322+0000","last_became_peered":"2024-03-09T00:04:13.927322+0000","last_unstale":"2024-03-09T00:06:06.921869+0000","last_undegraded":"2024-03-09T00:06:06.921869+0000","last_fullsized":"2024-03-09T00:06:06.921869+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:12:28.917211+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.d","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.271986+0000","last_change":"2024-03-09T00:04:18.031373+0000","last_active":"2024-03-09T00:06:06.271986+0000","last_peered":"2024-03-09T00:06:06.271986+0000","last_clean":"2024-03-09T00:06:06.271986+0000","last_became_active":"2024-03-09T00:04:18.030949+0000","last_became_peered":"2024-03-09T00:04:18.030949+0000","last_unstale":"2024-03-09T00:06:06.271986+0000","last_undegraded":"2024-03-09T00:06:06.271986+0000","last_fullsized":"2024-03-09T00:06:06.271986+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:05:08.077378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,2,10,3],"acting":[4,2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"2.b","version":"84'2","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082328+0000","last_change":"2024-03-09T00:04:13.333895+0000","last_active":"2024-03-09T00:06:06.082328+0000","last_peered":"2024-03-09T00:06:06.082328+0000","last_clean":"2024-03-09T00:06:06.082328+0000","last_became_active":"2024-03-09T00:04:13.333577+0000","last_became_peered":"2024-03-09T00:04:13.333577+0000","last_unstale":"2024-03-09T00:06:06.082328+0000","last_undegraded":"2024-03-09T00:06:06.082328+0000","last_fullsized":"2024-03-09T00:06:06.082328+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:01:27.614946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":1,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"91'5","reported_seq":34,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079588+0000","last_change":"2024-03-09T00:04:13.927171+0000","last_active":"2024-03-09T00:06:06.079588+0000","last_peered":"2024-03-09T00:06:06.079588+0000","last_clean":"2024-03-09T00:06:06.079588+0000","last_became_active":"2024-03-09T00:04:13.926844+0000","last_became_peered":"2024-03-09T00:04:13.926844+0000","last_unstale":"2024-03-09T00:06:06.079588+0000","last_undegraded":"2024-03-09T00:06:06.079588+0000","last_fullsized":"2024-03-09T00:06:06.079588+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:12:39.460061+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.a","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513345+0000","last_change":"2024-03-09T00:04:17.954803+0000","last_active":"2024-03-09T00:06:06.513345+0000","last_peered":"2024-03-09T00:06:06.513345+0000","last_clean":"2024-03-09T00:06:06.513345+0000","last_became_active":"2024-03-09T00:04:17.954323+0000","last_became_peered":"2024-03-09T00:04:17.954323+0000","last_unstale":"2024-03-09T00:06:06.513345+0000","last_undegraded":"2024-03-09T00:06:06.513345+0000","last_fullsized":"2024-03-09T00:06:06.513345+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:07:34.626146+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,6,7],"acting":[1,5,6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.c","version":"0'0","reported_seq":32,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081834+0000","last_change":"2024-03-09T00:04:13.335340+0000","last_active":"2024-03-09T00:06:06.081834+0000","last_peered":"2024-03-09T00:06:06.081834+0000","last_clean":"2024-03-09T00:06:06.081834+0000","last_became_active":"2024-03-09T00:04:13.334825+0000","last_became_peered":"2024-03-09T00:04:13.334825+0000","last_unstale":"2024-03-09T00:06:06.081834+0000","last_undegraded":"2024-03-09T00:06:06.081834+0000","last_fullsized":"2024-03-09T00:06:06.081834+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:51:35.778167+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079996+0000","last_change":"2024-03-09T00:04:13.929450+0000","last_active":"2024-03-09T00:06:06.079996+0000","last_peered":"2024-03-09T00:06:06.079996+0000","last_clean":"2024-03-09T00:06:06.079996+0000","last_became_active":"2024-03-09T00:04:13.929118+0000","last_became_peered":"2024-03-09T00:04:13.929118+0000","last_unstale":"2024-03-09T00:06:06.079996+0000","last_undegraded":"2024-03-09T00:06:06.079996+0000","last_fullsized":"2024-03-09T00:06:06.079996+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T10:39:55.378473+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.b","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081211+0000","last_change":"2024-03-09T00:04:17.959626+0000","last_active":"2024-03-09T00:06:06.081211+0000","last_peered":"2024-03-09T00:06:06.081211+0000","last_clean":"2024-03-09T00:06:06.081211+0000","last_became_active":"2024-03-09T00:04:17.959211+0000","last_became_peered":"2024-03-09T00:04:17.959211+0000","last_unstale":"2024-03-09T00:06:06.081211+0000","last_undegraded":"2024-03-09T00:06:06.081211+0000","last_fullsized":"2024-03-09T00:06:06.081211+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:42:25.212117+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,9,1,4],"acting":[0,9,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.d","version":"84'1","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076463+0000","last_change":"2024-03-09T00:04:13.331861+0000","last_active":"2024-03-09T00:06:06.076463+0000","last_peered":"2024-03-09T00:06:06.076463+0000","last_clean":"2024-03-09T00:06:06.076463+0000","last_became_active":"2024-03-09T00:04:13.329782+0000","last_became_peered":"2024-03-09T00:04:13.329782+0000","last_unstale":"2024-03-09T00:06:06.076463+0000","last_undegraded":"2024-03-09T00:06:06.076463+0000","last_fullsized":"2024-03-09T00:06:06.076463+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:07:53.062058+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079730+0000","last_change":"2024-03-09T00:04:13.930955+0000","last_active":"2024-03-09T00:06:06.079730+0000","last_peered":"2024-03-09T00:06:06.079730+0000","last_clean":"2024-03-09T00:06:06.079730+0000","last_became_active":"2024-03-09T00:04:13.930561+0000","last_became_peered":"2024-03-09T00:04:13.930561+0000","last_unstale":"2024-03-09T00:06:06.079730+0000","last_undegraded":"2024-03-09T00:06:06.079730+0000","last_fullsized":"2024-03-09T00:06:06.079730+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:38:27.390257+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.8","version":"0'0","reported_seq":23,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.075532+0000","last_change":"2024-03-09T00:04:17.956896+0000","last_active":"2024-03-09T00:06:06.075532+0000","last_peered":"2024-03-09T00:06:06.075532+0000","last_clean":"2024-03-09T00:06:06.075532+0000","last_became_active":"2024-03-09T00:04:17.956006+0000","last_became_peered":"2024-03-09T00:04:17.956006+0000","last_unstale":"2024-03-09T00:06:06.075532+0000","last_undegraded":"2024-03-09T00:06:06.075532+0000","last_fullsized":"2024-03-09T00:06:06.075532+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":87,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:16.916741+0000","last_clean_scrub_stamp":"2024-03-09T00:04:16.916741+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T02:53:47.910863+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7,1],"acting":[5,2,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.e","version":"84'2","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076859+0000","last_change":"2024-03-09T00:04:13.325396+0000","last_active":"2024-03-09T00:06:06.076859+0000","last_peered":"2024-03-09T00:06:06.076859+0000","last_clean":"2024-03-09T00:06:06.076859+0000","last_became_active":"2024-03-09T00:04:13.325024+0000","last_became_peered":"2024-03-09T00:04:13.325024+0000","last_unstale":"2024-03-09T00:06:06.076859+0000","last_undegraded":"2024-03-09T00:06:06.076859+0000","last_fullsized":"2024-03-09T00:06:06.076859+0000","mapping_epoch":82,"log_start":"0'0","ondisk_log_start":"0'0","created":82,"last_epoch_clean":83,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:12.308766+0000","last_clean_scrub_stamp":"2024-03-09T00:04:12.308766+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:10:03.560406+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.080006+0000","last_change":"2024-03-09T00:04:13.933899+0000","last_active":"2024-03-09T00:06:06.080006+0000","last_peered":"2024-03-09T00:06:06.080006+0000","last_clean":"2024-03-09T00:06:06.080006+0000","last_became_active":"2024-03-09T00:04:13.933582+0000","last_became_peered":"2024-03-09T00:04:13.933582+0000","last_unstale":"2024-03-09T00:06:06.080006+0000","last_undegraded":"2024-03-09T00:06:06.080006+0000","last_fullsized":"2024-03-09T00:06:06.080006+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T07:27:06.789899+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"91'9","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079554+0000","last_change":"2024-03-09T00:04:13.932256+0000","last_active":"2024-03-09T00:06:06.079554+0000","last_peered":"2024-03-09T00:06:06.079554+0000","last_clean":"2024-03-09T00:06:06.079554+0000","last_became_active":"2024-03-09T00:04:13.931918+0000","last_became_peered":"2024-03-09T00:04:13.931918+0000","last_unstale":"2024-03-09T00:06:06.079554+0000","last_undegraded":"2024-03-09T00:06:06.079554+0000","last_fullsized":"2024-03-09T00:06:06.079554+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T04:58:00.365216+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"91'12","reported_seq":42,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081561+0000","last_change":"2024-03-09T00:04:13.937155+0000","last_active":"2024-03-09T00:06:06.081561+0000","last_peered":"2024-03-09T00:06:06.081561+0000","last_clean":"2024-03-09T00:06:06.081561+0000","last_became_active":"2024-03-09T00:04:13.936849+0000","last_became_peered":"2024-03-09T00:04:13.936849+0000","last_unstale":"2024-03-09T00:06:06.081561+0000","last_undegraded":"2024-03-09T00:06:06.081561+0000","last_fullsized":"2024-03-09T00:06:06.081561+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:07:23.340522+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"91'8","reported_seq":38,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081789+0000","last_change":"2024-03-09T00:04:13.930184+0000","last_active":"2024-03-09T00:06:06.081789+0000","last_peered":"2024-03-09T00:06:06.081789+0000","last_clean":"2024-03-09T00:06:06.081789+0000","last_became_active":"2024-03-09T00:04:13.929876+0000","last_became_peered":"2024-03-09T00:04:13.929876+0000","last_unstale":"2024-03-09T00:06:06.081789+0000","last_undegraded":"2024-03-09T00:06:06.081789+0000","last_fullsized":"2024-03-09T00:06:06.081789+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:32:38.849978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"91'7","reported_seq":36,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.081489+0000","last_change":"2024-03-09T00:04:13.936352+0000","last_active":"2024-03-09T00:06:06.081489+0000","last_peered":"2024-03-09T00:06:06.081489+0000","last_clean":"2024-03-09T00:06:06.081489+0000","last_became_active":"2024-03-09T00:04:13.936042+0000","last_became_peered":"2024-03-09T00:04:13.936042+0000","last_unstale":"2024-03-09T00:06:06.081489+0000","last_undegraded":"2024-03-09T00:06:06.081489+0000","last_fullsized":"2024-03-09T00:06:06.081489+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:13:58.960982+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"91'4","reported_seq":33,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076818+0000","last_change":"2024-03-09T00:04:13.934331+0000","last_active":"2024-03-09T00:06:06.076818+0000","last_peered":"2024-03-09T00:06:06.076818+0000","last_clean":"2024-03-09T00:06:06.076818+0000","last_became_active":"2024-03-09T00:04:13.933978+0000","last_became_peered":"2024-03-09T00:04:13.933978+0000","last_unstale":"2024-03-09T00:06:06.076818+0000","last_undegraded":"2024-03-09T00:06:06.076818+0000","last_fullsized":"2024-03-09T00:06:06.076818+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:12:38.585382+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079815+0000","last_change":"2024-03-09T00:04:13.939945+0000","last_active":"2024-03-09T00:06:06.079815+0000","last_peered":"2024-03-09T00:06:06.079815+0000","last_clean":"2024-03-09T00:06:06.079815+0000","last_became_active":"2024-03-09T00:04:13.938986+0000","last_became_peered":"2024-03-09T00:04:13.938986+0000","last_unstale":"2024-03-09T00:06:06.079815+0000","last_undegraded":"2024-03-09T00:06:06.079815+0000","last_fullsized":"2024-03-09T00:06:06.079815+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T09:58:25.290429+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"91'15","reported_seq":45,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.082078+0000","last_change":"2024-03-09T00:04:13.931890+0000","last_active":"2024-03-09T00:06:06.082078+0000","last_peered":"2024-03-09T00:06:06.082078+0000","last_clean":"2024-03-09T00:06:06.082078+0000","last_became_active":"2024-03-09T00:04:13.931571+0000","last_became_peered":"2024-03-09T00:04:13.931571+0000","last_unstale":"2024-03-09T00:06:06.082078+0000","last_undegraded":"2024-03-09T00:06:06.082078+0000","last_fullsized":"2024-03-09T00:06:06.082078+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T05:52:56.397551+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"91'2","reported_seq":31,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921924+0000","last_change":"2024-03-09T00:04:13.932681+0000","last_active":"2024-03-09T00:06:06.921924+0000","last_peered":"2024-03-09T00:06:06.921924+0000","last_clean":"2024-03-09T00:06:06.921924+0000","last_became_active":"2024-03-09T00:04:13.932127+0000","last_became_peered":"2024-03-09T00:04:13.932127+0000","last_unstale":"2024-03-09T00:06:06.921924+0000","last_undegraded":"2024-03-09T00:06:06.921924+0000","last_fullsized":"2024-03-09T00:06:06.921924+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T11:38:01.169666+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"91'6","reported_seq":35,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076018+0000","last_change":"2024-03-09T00:04:13.928511+0000","last_active":"2024-03-09T00:06:06.076018+0000","last_peered":"2024-03-09T00:06:06.076018+0000","last_clean":"2024-03-09T00:06:06.076018+0000","last_became_active":"2024-03-09T00:04:13.928193+0000","last_became_peered":"2024-03-09T00:04:13.928193+0000","last_unstale":"2024-03-09T00:06:06.076018+0000","last_undegraded":"2024-03-09T00:06:06.076018+0000","last_fullsized":"2024-03-09T00:06:06.076018+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T03:54:21.365691+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.921769+0000","last_change":"2024-03-09T00:04:13.928676+0000","last_active":"2024-03-09T00:06:06.921769+0000","last_peered":"2024-03-09T00:06:06.921769+0000","last_clean":"2024-03-09T00:06:06.921769+0000","last_became_active":"2024-03-09T00:04:13.928080+0000","last_became_peered":"2024-03-09T00:04:13.928080+0000","last_unstale":"2024-03-09T00:06:06.921769+0000","last_undegraded":"2024-03-09T00:06:06.921769+0000","last_fullsized":"2024-03-09T00:06:06.921769+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:58:42.000650+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"91'10","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.513579+0000","last_change":"2024-03-09T00:04:13.938425+0000","last_active":"2024-03-09T00:06:06.513579+0000","last_peered":"2024-03-09T00:06:06.513579+0000","last_clean":"2024-03-09T00:06:06.513579+0000","last_became_active":"2024-03-09T00:04:13.938030+0000","last_became_peered":"2024-03-09T00:04:13.938030+0000","last_unstale":"2024-03-09T00:06:06.513579+0000","last_undegraded":"2024-03-09T00:06:06.513579+0000","last_fullsized":"2024-03-09T00:06:06.513579+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:22:38.825036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"91'11","reported_seq":40,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.076412+0000","last_change":"2024-03-09T00:04:13.927690+0000","last_active":"2024-03-09T00:06:06.076412+0000","last_peered":"2024-03-09T00:06:06.076412+0000","last_clean":"2024-03-09T00:06:06.076412+0000","last_became_active":"2024-03-09T00:04:13.927381+0000","last_became_peered":"2024-03-09T00:04:13.927381+0000","last_unstale":"2024-03-09T00:06:06.076412+0000","last_undegraded":"2024-03-09T00:06:06.076412+0000","last_fullsized":"2024-03-09T00:06:06.076412+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T08:46:03.256908+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"91'8","reported_seq":37,"reported_epoch":93,"state":"active+clean","last_fresh":"2024-03-09T00:06:06.079497+0000","last_change":"2024-03-09T00:04:13.936250+0000","last_active":"2024-03-09T00:06:06.079497+0000","last_peered":"2024-03-09T00:06:06.079497+0000","last_clean":"2024-03-09T00:06:06.079497+0000","last_became_active":"2024-03-09T00:04:13.935877+0000","last_became_peered":"2024-03-09T00:04:13.935877+0000","last_unstale":"2024-03-09T00:06:06.079497+0000","last_undegraded":"2024-03-09T00:06:06.079497+0000","last_fullsized":"2024-03-09T00:06:06.079497+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-09T00:04:13.310345+0000","last_clean_scrub_stamp":"2024-03-09T00:04:13.310345+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-10T00:56:22.552737+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":32,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":32768,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":2,"ondisk_log_size":2,"up":128,"acting":128,"num_store_stats":12},{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":500,"ondisk_log_size":500,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":136,"num_read_kb":119,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4157440,"data_stored":4132576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70084,"num_objects":22,"num_object_clones":0,"num_object_copies":66,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":258,"num_read_kb":200,"num_write":89,"num_write_kb":102,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":286720,"data_stored":199198,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":76,"ondisk_log_size":76,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":80,"seq":343597383723,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36368,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708784,"statfs":{"total":95995035648,"available":95957794816,"internally_reserved":0,"allocated":1052672,"data_stored":519110,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":36174276},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":72,"seq":309237645361,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32868,"kb_used_data":1692,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712284,"statfs":{"total":95995035648,"available":95961378816,"internally_reserved":0,"allocated":1732608,"data_stored":1174862,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":67,"seq":287762808886,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36380,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708772,"statfs":{"total":95995035648,"available":95957782528,"internally_reserved":0,"allocated":1060864,"data_stored":523788,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":36174272},"hb_peers":[0,1,2,3,4,5,7,8,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":59,"seq":253403070523,"num_pgs":31,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1036,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1060864,"data_stored":519714,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":51,"seq":219043332159,"num_pgs":37,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36952,"kb_used_data":1616,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708200,"statfs":{"total":95995035648,"available":95957196800,"internally_reserved":0,"allocated":1654784,"data_stored":1114156,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1590,"internal_metadata":36174282},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":46,"seq":197568495685,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32208,"kb_used_data":1024,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712944,"statfs":{"total":95995035648,"available":95962054656,"internally_reserved":0,"allocated":1048576,"data_stored":515014,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":31914438},"hb_peers":[0,1,2,3,4,5,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574942,"num_pgs":33,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36432,"kb_used_data":1096,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708720,"statfs":{"total":95995035648,"available":95957729280,"internally_reserved":0,"allocated":1122304,"data_stored":578927,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738467,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28052,"kb_used_data":1028,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717100,"statfs":{"total":95995035648,"available":95966310400,"internally_reserved":0,"allocated":1052672,"data_stored":515104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411417,"num_pgs":35,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28124,"kb_used_data":1100,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717028,"statfs":{"total":95995035648,"available":95966236672,"internally_reserved":0,"allocated":1126400,"data_stored":578653,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":25,"seq":107374182483,"num_pgs":39,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":515118,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":27654599},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":33,"seq":141733920847,"num_pgs":36,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36376,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708776,"statfs":{"total":95995035648,"available":95957786624,"internally_reserved":0,"allocated":1056768,"data_stored":519132,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":38,"seq":163208757322,"num_pgs":44,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36960,"kb_used_data":1620,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708192,"statfs":{"total":95995035648,"available":95957188608,"internally_reserved":0,"allocated":1658880,"data_stored":1109660,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":90,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":63913,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":77824,"data_stored":63639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":22,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":182,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":604,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":582,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":90112,"data_stored":69480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":8192,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":4,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":4096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-09T00:06:41.215 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:41.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:41 smithi179 ceph-mon[36637]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-09T00:06:41.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:41 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1998829124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 -- 172.21.15.73:0/4250601896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc104720 msgr2=0x7f5edc106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 --2- 172.21.15.73:0/4250601896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc104720 0x7f5edc106b10 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009900 tx=0x7f5ec402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 -- 172.21.15.73:0/4250601896 shutdown_connections 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 --2- 172.21.15.73:0/4250601896 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 --2- 172.21.15.73:0/4250601896 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc104720 0x7f5edc106b10 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 --2- 172.21.15.73:0/4250601896 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5edc101df0 0x7f5edc1041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 -- 172.21.15.73:0/4250601896 >> 172.21.15.73:0/4250601896 conn(0x7f5edc0fb7a0 msgr2=0x7f5edc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 -- 172.21.15.73:0/4250601896 shutdown_connections 2024-03-09T00:06:41.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.358+0000 7f5ee088e640 1 -- 172.21.15.73:0/4250601896 wait complete. 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 Processor -- start 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 -- start start 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5edc104720 0x7f5edc197b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc19ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc10cff0 con 0x7f5edc101df0 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc10ce70 con 0x7f5edc104720 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5ee088e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5edc10d170 con 0x7f5edc107050 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60246/0 (socket says 172.21.15.73:60246) 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 -- 172.21.15.73:0/2421240209 learned_addr learned my addr 172.21.15.73:0/2421240209 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:41.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5edad9d640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc19ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 -- 172.21.15.73:0/2421240209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 msgr2=0x7f5edc19ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc19ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 -- 172.21.15.73:0/2421240209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5edc104720 msgr2=0x7f5edc197b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5edc104720 0x7f5edc197b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5eda59c640 1 -- 172.21.15.73:0/2421240209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc19f260 con 0x7f5edc101df0 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.359+0000 7f5edad9d640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5eda59c640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00d950 tx=0x7f5ecc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc017070 con 0x7f5edc101df0 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5edc19f550 con 0x7f5edc101df0 2024-03-09T00:06:41.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ecc009c30 con 0x7f5edc101df0 2024-03-09T00:06:41.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ecc00f6b0 con 0x7f5edc101df0 2024-03-09T00:06:41.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.360+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5edc19faa0 con 0x7f5edc101df0 2024-03-09T00:06:41.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.361+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5edc0fd290 con 0x7f5edc101df0 2024-03-09T00:06:41.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.362+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5ecc0109d0 con 0x7f5edc101df0 2024-03-09T00:06:41.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.363+0000 7f5ed37fe640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 0x7f5ec80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.363+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5ecc0cead0 con 0x7f5edc101df0 2024-03-09T00:06:41.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.363+0000 7f5ed3fff640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 0x7f5ec80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.363+0000 7f5ed3fff640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 0x7f5ec80a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009230 tx=0x7f5ec4000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.366+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5ecc009de0 con 0x7f5edc101df0 2024-03-09T00:06:41.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:41 smithi102 ceph-mon[36106]: pgmap v371: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 767 B/s wr, 3 op/s 2024-03-09T00:06:41.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:41 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1998829124' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:41.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.545+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5edc02d120 con 0x7f5edc101df0 2024-03-09T00:06:41.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.546+0000 7f5ed37fe640 1 -- 172.21.15.73:0/2421240209 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7f5ecc09b7c0 con 0x7f5edc101df0 2024-03-09T00:06:41.541 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:41.541 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:41.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 msgr2=0x7f5ec80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 0x7f5ec80a4cc0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f5ec4009230 tx=0x7f5ec4000bc0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 msgr2=0x7f5edc197640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7f5ecc00d950 tx=0x7f5ecc00de10 comp rx=0 tx=0).stop 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 shutdown_connections 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ec80a2800 0x7f5ec80a4cc0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5edc107050 0x7f5edc19ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5edc104720 0x7f5edc197b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 --2- 172.21.15.73:0/2421240209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5edc101df0 0x7f5edc197640 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 >> 172.21.15.73:0/2421240209 conn(0x7f5edc0fb7a0 msgr2=0x7f5edc1026b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 shutdown_connections 2024-03-09T00:06:41.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.548+0000 7f5ee088e640 1 -- 172.21.15.73:0/2421240209 wait complete. 2024-03-09T00:06:41.555 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 5, 'flags_names': 'hashpspool,ec_overwrites', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:06:41.555 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.665+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1737867275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78101dd0 msgr2=0x7f6c781041c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.665+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1737867275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78101dd0 0x7f6c781041c0 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c009900 tx=0x7f6c6c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1737867275 shutdown_connections 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1737867275 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78107030 0x7f6c78109420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1737867275 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78104700 0x7f6c78106af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1737867275 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78101dd0 0x7f6c781041c0 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1737867275 >> 172.21.15.73:0/1737867275 conn(0x7f6c780fb780 msgr2=0x7f6c780fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1737867275 shutdown_connections 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1737867275 wait complete. 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 Processor -- start 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- start start 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 0x7f6c78197610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78107030 0x7f6c7819ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7810cf80 con 0x7f6c78104700 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7810ce00 con 0x7f6c78107030 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c7ecd7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7810d100 con 0x7f6c78101dd0 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c77fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60262/0 (socket says 172.21.15.73:60262) 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.666+0000 7f6c77fff640 1 -- 172.21.15.73:0/1508663514 learned_addr learned my addr 172.21.15.73:0/1508663514 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c7d274640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78107030 0x7f6c7819ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c7ca73640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 0x7f6c78197610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 -- 172.21.15.73:0/1508663514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 msgr2=0x7f6c78197610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 0x7f6c78197610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 -- 172.21.15.73:0/1508663514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78107030 msgr2=0x7f6c7819ebc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78107030 0x7f6c7819ebc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 -- 172.21.15.73:0/1508663514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7819f230 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c7ca73640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 0x7f6c78197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c77fff640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005fd0 tx=0x7f6c6400dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64017070 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7819f520 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c64009c30 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c64010460 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.667+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7819fa70 con 0x7f6c78104700 2024-03-09T00:06:41.665 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.668+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c780fd270 con 0x7f6c78104700 2024-03-09T00:06:41.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.669+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f6c64003e70 con 0x7f6c78104700 2024-03-09T00:06:41.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.670+0000 7f6c75ffb640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 0x7f6c680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.670+0000 7f6c7ca73640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 0x7f6c680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.670+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f6c640cebf0 con 0x7f6c78104700 2024-03-09T00:06:41.666 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.670+0000 7f6c7ca73640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 0x7f6c680a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0096c0 tx=0x7f6c6c000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.674+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6c64020030 con 0x7f6c78104700 2024-03-09T00:06:41.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.848+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c78061ef0 con 0x7f6c78104700 2024-03-09T00:06:41.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.849+0000 7f6c75ffb640 1 -- 172.21.15.73:0/1508663514 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v93) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c6409b8e0 con 0x7f6c78104700 2024-03-09T00:06:41.843 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 msgr2=0x7f6c680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 0x7f6c680a4cc0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f6c6c0096c0 tx=0x7f6c6c000bc0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 msgr2=0x7f6c78197b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f6c64005fd0 tx=0x7f6c6400dde0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 shutdown_connections 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f6c680a2800 0x7f6c680a4cc0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6c78107030 0x7f6c7819ebc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6c78104700 0x7f6c78197b50 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 --2- 172.21.15.73:0/1508663514 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6c78101dd0 0x7f6c78197610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 >> 172.21.15.73:0/1508663514 conn(0x7f6c780fb780 msgr2=0x7f6c78102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 shutdown_connections 2024-03-09T00:06:41.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.851+0000 7f6c7ecd7640 1 -- 172.21.15.73:0/1508663514 wait complete. 2024-03-09T00:06:41.857 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:06:41.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 -- 172.21.15.73:0/2238102393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102ed0 msgr2=0x7ff678109760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 --2- 172.21.15.73:0/2238102393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102ed0 0x7ff678109760 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7ff670009ca0 tx=0x7ff67002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 -- 172.21.15.73:0/2238102393 shutdown_connections 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 --2- 172.21.15.73:0/2238102393 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102ed0 0x7ff678109760 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 --2- 172.21.15.73:0/2238102393 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678102510 0x7ff678102990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 --2- 172.21.15.73:0/2238102393 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678101320 0x7ff678101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 -- 172.21.15.73:0/2238102393 >> 172.21.15.73:0/2238102393 conn(0x7ff6780fc890 msgr2=0x7ff6780fecd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 -- 172.21.15.73:0/2238102393 shutdown_connections 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.969+0000 7ff67cca4640 1 -- 172.21.15.73:0/2238102393 wait complete. 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 Processor -- start 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 -- start start 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 0x7ff6781975b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678102ed0 0x7ff67819eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67810cfb0 con 0x7ff678102510 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67810ce30 con 0x7ff678101320 2024-03-09T00:06:41.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67cca4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff67810d130 con 0x7ff678102ed0 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60284/0 (socket says 172.21.15.73:60284) 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 -- 172.21.15.73:0/980500950 learned_addr learned my addr 172.21.15.73:0/980500950 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff676d9d640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678102ed0 0x7ff67819eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67659c640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 0x7ff6781975b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 -- 172.21.15.73:0/980500950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678102ed0 msgr2=0x7ff67819eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678102ed0 0x7ff67819eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 -- 172.21.15.73:0/980500950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 msgr2=0x7ff6781975b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 0x7ff6781975b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff66ffff640 1 -- 172.21.15.73:0/980500950 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff67819f1d0 con 0x7ff678102510 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.970+0000 7ff67659c640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 0x7ff6781975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff66ffff640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff66000e9f0 tx=0x7ff66000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff66000cd80 con 0x7ff678102510 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff660010910 con 0x7ff678102510 2024-03-09T00:06:41.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff67819f4c0 con 0x7ff678102510 2024-03-09T00:06:41.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff6600188f0 con 0x7ff678102510 2024-03-09T00:06:41.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.971+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67819fa10 con 0x7ff678102510 2024-03-09T00:06:41.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.972+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6781085e0 con 0x7ff678102510 2024-03-09T00:06:41.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.973+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7ff660010ac0 con 0x7ff678102510 2024-03-09T00:06:41.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.973+0000 7ff66f7fe640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 0x7ff6640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:41.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.974+0000 7ff67659c640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 0x7ff6640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:41.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.974+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7ff6600cf630 con 0x7ff678102510 2024-03-09T00:06:41.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.974+0000 7ff67659c640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 0x7ff6640a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff678102370 tx=0x7ff668005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:41.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:41.977+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff66009c320 con 0x7ff678102510 2024-03-09T00:06:42.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.153+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff67802d0c0 con 0x7ff678102510 2024-03-09T00:06:42.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.153+0000 7ff66f7fe640 1 -- 172.21.15.73:0/980500950 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v93) v1 ==== 104+0+11 (secure 0 0 0) 0x7ff660086020 con 0x7ff678102510 2024-03-09T00:06:42.148 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:42.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 msgr2=0x7ff6640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 0x7ff6640a4cc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff678102370 tx=0x7ff668005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:42.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 msgr2=0x7ff678197af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7ff66000e9f0 tx=0x7ff66000eeb0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 shutdown_connections 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7ff6640a2800 0x7ff6640a4cc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff678102ed0 0x7ff67819eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff678102510 0x7ff678197af0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 --2- 172.21.15.73:0/980500950 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff678101320 0x7ff6781975b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 >> 172.21.15.73:0/980500950 conn(0x7ff6780fc890 msgr2=0x7ff6781057b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 shutdown_connections 2024-03-09T00:06:42.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.155+0000 7ff67cca4640 1 -- 172.21.15.73:0/980500950 wait complete. 2024-03-09T00:06:42.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:42 smithi073 ceph-mon[32309]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:42 smithi073 ceph-mon[32309]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:42 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2421240209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:42.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:42 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1508663514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:42.161 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 -- 172.21.15.73:0/1313334834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 msgr2=0x7f67a8100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/1313334834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a8100620 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f67a0009900 tx=0x7f67a002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 -- 172.21.15.73:0/1313334834 shutdown_connections 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/1313334834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 0x7f67a8108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/1313334834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 0x7f67a8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/1313334834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a8100620 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 -- 172.21.15.73:0/1313334834 >> 172.21.15.73:0/1313334834 conn(0x7f67a80fb790 msgr2=0x7f67a80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 -- 172.21.15.73:0/1313334834 shutdown_connections 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.272+0000 7f67ac8a4640 1 -- 172.21.15.73:0/1313334834 wait complete. 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 Processor -- start 2024-03-09T00:06:42.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 -- start start 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 0x7f67a8196a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 0x7f67a819da70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a810be90 con 0x7f67a8100200 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a810bd10 con 0x7f67a8101db0 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67a810c010 con 0x7f67a81013f0 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60302/0 (socket says 172.21.15.73:60302) 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 -- 172.21.15.73:0/129816780 learned_addr learned my addr 172.21.15.73:0/129816780 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a6d9d640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 0x7f67a819da70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a5d9b640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 0x7f67a8196a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 -- 172.21.15.73:0/129816780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 msgr2=0x7f67a8196a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 0x7f67a8196a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 -- 172.21.15.73:0/129816780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 msgr2=0x7f67a819da70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 0x7f67a819da70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 -- 172.21.15.73:0/129816780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a819e0e0 con 0x7f67a8100200 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67a659c640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67a0005ab0 tx=0x7f67a0005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0041070 con 0x7f67a8100200 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.273+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67a819e370 con 0x7f67a8100200 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.274+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67a00054e0 con 0x7f67a8100200 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.274+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67a0034440 con 0x7f67a8100200 2024-03-09T00:06:42.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.274+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67a819e8b0 con 0x7f67a8100200 2024-03-09T00:06:42.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.274+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67a8107540 con 0x7f67a8100200 2024-03-09T00:06:42.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.275+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f67a0044070 con 0x7f67a8100200 2024-03-09T00:06:42.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.276+0000 7f679f7fe640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 0x7f67940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.276+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f67a00f0b00 con 0x7f67a8100200 2024-03-09T00:06:42.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.276+0000 7f67a5d9b640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 0x7f67940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.276+0000 7f67a5d9b640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 0x7f67940a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6790005e10 tx=0x7f6790005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.280+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f67a00bd7f0 con 0x7f67a8100200 2024-03-09T00:06:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:42 smithi179 ceph-mon[36637]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:42 smithi179 ceph-mon[36637]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:42 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2421240209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:42.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:42 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1508663514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:42.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.453+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f67a8061ef0 con 0x7f67a8100200 2024-03-09T00:06:42.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.453+0000 7f679f7fe640 1 -- 172.21.15.73:0/129816780 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v93) v1 ==== 100+0+11 (secure 0 0 0) 0x7f67a00bd0f0 con 0x7f67a8100200 2024-03-09T00:06:42.448 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 msgr2=0x7f67940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 0x7f67940a4cc0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f6790005e10 tx=0x7f6790005da0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 msgr2=0x7f67a81964c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f67a0005ab0 tx=0x7f67a0005dc0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 shutdown_connections 2024-03-09T00:06:42.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f67940a2800 0x7f67940a4cc0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f67a8101db0 0x7f67a819da70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f67a81013f0 0x7f67a8196a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 --2- 172.21.15.73:0/129816780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f67a8100200 0x7f67a81964c0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.455+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 >> 172.21.15.73:0/129816780 conn(0x7f67a80fb790 msgr2=0x7f67a8104710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.456+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 shutdown_connections 2024-03-09T00:06:42.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.456+0000 7f67ac8a4640 1 -- 172.21.15.73:0/129816780 wait complete. 2024-03-09T00:06:42.461 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:06:42.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:42 smithi102 ceph-mon[36106]: from='client.15450 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:42 smithi102 ceph-mon[36106]: from='client.15456 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:42.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:42 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2421240209' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:42.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:42 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1508663514' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 -- 172.21.15.73:0/3835778357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 msgr2=0x7f30b8100620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 --2- 172.21.15.73:0/3835778357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b8100620 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f30ac009900 tx=0x7f30ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 -- 172.21.15.73:0/3835778357 shutdown_connections 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 --2- 172.21.15.73:0/3835778357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b8101db0 0x7f30b8108640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 --2- 172.21.15.73:0/3835778357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b81013f0 0x7f30b8101870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 --2- 172.21.15.73:0/3835778357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b8100620 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 -- 172.21.15.73:0/3835778357 >> 172.21.15.73:0/3835778357 conn(0x7f30b80fb790 msgr2=0x7f30b80fdbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 -- 172.21.15.73:0/3835778357 shutdown_connections 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 -- 172.21.15.73:0/3835778357 wait complete. 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.571+0000 7f30bece8640 1 Processor -- start 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 -- start start 2024-03-09T00:06:42.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 0x7f30b819c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 0x7f30b819ea00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b810be90 con 0x7f30b8100200 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b810bd10 con 0x7f30b81013f0 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30b810c010 con 0x7f30b8101db0 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60310/0 (socket says 172.21.15.73:60310) 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 -- 172.21.15.73:0/3420449304 learned_addr learned my addr 172.21.15.73:0/3420449304 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30b7fff640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 0x7f30b819c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bd285640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 0x7f30b819ea00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 -- 172.21.15.73:0/3420449304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 msgr2=0x7f30b819ea00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 0x7f30b819ea00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 -- 172.21.15.73:0/3420449304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 msgr2=0x7f30b819c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 0x7f30b819c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 -- 172.21.15.73:0/3420449304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b819f2c0 con 0x7f30b8100200 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bd285640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 0x7f30b819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30b7fff640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 0x7f30b819c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bca84640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f30ac000c00 tx=0x7f30ac004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac041070 con 0x7f30b8100200 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f30ac0054e0 con 0x7f30b8100200 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f30ac0346a0 con 0x7f30b8100200 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.572+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30b819f550 con 0x7f30b8100200 2024-03-09T00:06:42.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.573+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30b819fa40 con 0x7f30b8100200 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.574+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30b81075c0 con 0x7f30b8100200 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.575+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f30ac044070 con 0x7f30b8100200 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.575+0000 7f30b5ffb640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 0x7f30a80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.575+0000 7f30b7fff640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 0x7f30a80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.575+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f30ac0f0e60 con 0x7f30b8100200 2024-03-09T00:06:42.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.576+0000 7f30b7fff640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 0x7f30a80a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30a4005fd0 tx=0x7f30a4005ea0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.579+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f30ac0bdb50 con 0x7f30b8100200 2024-03-09T00:06:42.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.750+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f30b802d0c0 con 0x7f30b8100200 2024-03-09T00:06:42.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.751+0000 7f30b5ffb640 1 -- 172.21.15.73:0/3420449304 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v93) v1 ==== 103+0+11 (secure 0 0 0) 0x7f30ac003e20 con 0x7f30b8100200 2024-03-09T00:06:42.746 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 msgr2=0x7f30a80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 0x7f30a80a4cc0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f30a4005fd0 tx=0x7f30a4005ea0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 msgr2=0x7f30b806a470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7f30ac000c00 tx=0x7f30ac004040 comp rx=0 tx=0).stop 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 shutdown_connections 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f30a80a2800 0x7f30a80a4cc0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30b8101db0 0x7f30b819ea00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30b81013f0 0x7f30b819c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 --2- 172.21.15.73:0/3420449304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f30b8100200 0x7f30b806a470 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 >> 172.21.15.73:0/3420449304 conn(0x7f30b80fb790 msgr2=0x7f30b8104790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 shutdown_connections 2024-03-09T00:06:42.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.753+0000 7f30bece8640 1 -- 172.21.15.73:0/3420449304 wait complete. 2024-03-09T00:06:42.759 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:06:42.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 -- 172.21.15.73:0/2026063758 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 msgr2=0x7fbc20101010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2026063758 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc20101010 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009900 tx=0x7fbc1002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:42.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 -- 172.21.15.73:0/2026063758 shutdown_connections 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2026063758 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc20101550 0x7fbc20107de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2026063758 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc20101010 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2026063758 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc200ff9a0 0x7fbc200ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 -- 172.21.15.73:0/2026063758 >> 172.21.15.73:0/2026063758 conn(0x7fbc200faf10 msgr2=0x7fbc200fd350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 -- 172.21.15.73:0/2026063758 shutdown_connections 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.866+0000 7fbc27d24640 1 -- 172.21.15.73:0/2026063758 wait complete. 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 Processor -- start 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 -- start start 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc200ff9a0 0x7fbc200fcf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20101550 0x7fbc2019e9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2010b680 con 0x7fbc20100b90 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2010b500 con 0x7fbc200ff9a0 2024-03-09T00:06:42.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc27d24640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc2010b800 con 0x7fbc20101550 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60326/0 (socket says 172.21.15.73:60326) 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 -- 172.21.15.73:0/2924942682 learned_addr learned my addr 172.21.15.73:0/2924942682 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc262c1640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20101550 0x7fbc2019e9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 -- 172.21.15.73:0/2924942682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20101550 msgr2=0x7fbc2019e9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc25ac0640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc200ff9a0 0x7fbc200fcf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20101550 0x7fbc2019e9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 -- 172.21.15.73:0/2924942682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc200ff9a0 msgr2=0x7fbc200fcf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc200ff9a0 0x7fbc200fcf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.867+0000 7fbc252bf640 1 -- 172.21.15.73:0/2924942682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc2019f270 con 0x7fbc20100b90 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc252bf640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009230 tx=0x7fbc10004040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10041070 con 0x7fbc20100b90 2024-03-09T00:06:42.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc100054e0 con 0x7fbc20100b90 2024-03-09T00:06:42.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc2019f500 con 0x7fbc20100b90 2024-03-09T00:06:42.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc100346a0 con 0x7fbc20100b90 2024-03-09T00:06:42.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.868+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc2019f9f0 con 0x7fbc20100b90 2024-03-09T00:06:42.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.869+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc2019fc50 con 0x7fbc20100b90 2024-03-09T00:06:42.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.870+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fbc10044070 con 0x7fbc20100b90 2024-03-09T00:06:42.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.870+0000 7fbc16ffd640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 0x7fbc0c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:42.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.870+0000 7fbc25ac0640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 0x7fbc0c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:42.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.870+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fbc100f0f70 con 0x7fbc20100b90 2024-03-09T00:06:42.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.871+0000 7fbc25ac0640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 0x7fbc0c0a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c005fd0 tx=0x7fbc1c005ce0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:42.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:42.874+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbc100bdc60 con 0x7fbc20100b90 2024-03-09T00:06:43.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.088+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbc20107090 con 0x7fbc20100b90 2024-03-09T00:06:43.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.088+0000 7fbc16ffd640 1 -- 172.21.15.73:0/2924942682 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fbc100bd560 con 0x7fbc20100b90 2024-03-09T00:06:43.083 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:43.084 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:06:43.084 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:06:43.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.090+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 msgr2=0x7fbc0c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.090+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 0x7fbc0c0a4cc0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7fbc1c005fd0 tx=0x7fbc1c005ce0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.090+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 msgr2=0x7fbc2019c3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.090+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7fbc10009230 tx=0x7fbc10004040 comp rx=0 tx=0).stop 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 shutdown_connections 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fbc0c0a2800 0x7fbc0c0a4cc0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbc20101550 0x7fbc2019e9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbc20100b90 0x7fbc2019c3e0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 --2- 172.21.15.73:0/2924942682 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbc200ff9a0 0x7fbc200fcf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 >> 172.21.15.73:0/2924942682 conn(0x7fbc200faf10 msgr2=0x7fbc20104260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 shutdown_connections 2024-03-09T00:06:43.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.091+0000 7fbc27d24640 1 -- 172.21.15.73:0/2924942682 wait complete. 2024-03-09T00:06:43.097 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:06:43.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 -- 172.21.15.73:0/743955657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0101dc0 msgr2=0x7fc7d0108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 --2- 172.21.15.73:0/743955657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0101dc0 0x7fc7d0108650 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fc7c8009ca0 tx=0x7fc7c802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 -- 172.21.15.73:0/743955657 shutdown_connections 2024-03-09T00:06:43.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 --2- 172.21.15.73:0/743955657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0101dc0 0x7fc7d0108650 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 --2- 172.21.15.73:0/743955657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 0x7fc7d0101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 --2- 172.21.15.73:0/743955657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d0100210 0x7fc7d0100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 -- 172.21.15.73:0/743955657 >> 172.21.15.73:0/743955657 conn(0x7fc7d00fb7a0 msgr2=0x7fc7d00fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 -- 172.21.15.73:0/743955657 shutdown_connections 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.204+0000 7fc7d5832640 1 -- 172.21.15.73:0/743955657 wait complete. 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 Processor -- start 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 -- start start 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 0x7fc7d019c330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d0101dc0 0x7fc7d019e970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d010bee0 con 0x7fc7d0100210 2024-03-09T00:06:43.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d010bd60 con 0x7fc7d0101dc0 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7d010c060 con 0x7fc7d0101400 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60348/0 (socket says 172.21.15.73:60348) 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 -- 172.21.15.73:0/3604130425 learned_addr learned my addr 172.21.15.73:0/3604130425 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7c7fff640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 0x7fc7d019c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 -- 172.21.15.73:0/3604130425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 msgr2=0x7fc7d019c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 0x7fc7d019c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 -- 172.21.15.73:0/3604130425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d0101dc0 msgr2=0x7fc7d019e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d0101dc0 0x7fc7d019e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 -- 172.21.15.73:0/3604130425 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7d019f230 con 0x7fc7d0100210 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7ceffd640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c970 tx=0x7fc7c000ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c0004220 con 0x7fc7d0100210 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc7c00043c0 con 0x7fc7d0100210 2024-03-09T00:06:43.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7d019f520 con 0x7fc7d0100210 2024-03-09T00:06:43.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.205+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc7c000f450 con 0x7fc7d0100210 2024-03-09T00:06:43.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.206+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7d019fa70 con 0x7fc7d0100210 2024-03-09T00:06:43.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.207+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7d0107530 con 0x7fc7d0100210 2024-03-09T00:06:43.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.208+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fc7c00079b0 con 0x7fc7d0100210 2024-03-09T00:06:43.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.208+0000 7fc7d4830640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.209+0000 7fc7c7fff640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.209+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fc7c00cec00 con 0x7fc7d0100210 2024-03-09T00:06:43.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.209+0000 7fc7c7fff640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc7d006a240 tx=0x7fc7b8008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.213+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc7c00d5080 con 0x7fc7d0100210 2024-03-09T00:06:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:43 smithi179 ceph-mon[36637]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:43 smithi179 ceph-mon[36637]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-09T00:06:43.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/980500950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/129816780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:43 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3420449304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.388+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc7d0061f90 con 0x7fc7d0100210 2024-03-09T00:06:43.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.389+0000 7fc7d4830640 1 -- 172.21.15.73:0/3604130425 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v93) v1 ==== 74+0+24783 (secure 0 0 0) 0x7fc7c0014030 con 0x7fc7d0100210 2024-03-09T00:06:43.384 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:06:43.385 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":93,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:06:06.068347+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":5,"flags_names":"hashpspool,ec_overwrites","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:06:43.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 msgr2=0x7fc7bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fc7d006a240 tx=0x7fc7b8008040 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 msgr2=0x7fc7d006a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7fc7c000c970 tx=0x7fc7c000ce30 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 shutdown_connections 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fc7bc0a2800 0x7fc7bc0a4cc0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc7d0101dc0 0x7fc7d019e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc7d0101400 0x7fc7d019c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 --2- 172.21.15.73:0/3604130425 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc7d0100210 0x7fc7d006a4d0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 >> 172.21.15.73:0/3604130425 conn(0x7fc7d00fb7a0 msgr2=0x7fc7d0104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 shutdown_connections 2024-03-09T00:06:43.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.391+0000 7fc7d5832640 1 -- 172.21.15.73:0/3604130425 wait complete. 2024-03-09T00:06:43.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:43 smithi073 ceph-mon[32309]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:43.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:43 smithi073 ceph-mon[32309]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-09T00:06:43.396 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/980500950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.397 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/129816780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.397 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:43 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3420449304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.398 DEBUG:tasks.fwd_scrub:Yielding 2024-03-09T00:06:43.398 INFO:teuthology.run_tasks:Running task exec... 2024-03-09T00:06:43.412 INFO:teuthology.task.exec:Executing custom commands... 2024-03-09T00:06:43.412 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:06:43.412 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph mgr module enable snap_schedule' 2024-03-09T00:06:43.412 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-09T00:06:43.412 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:06:43.413 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:06:43.413 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:06:43.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:43 smithi102 ceph-mon[36106]: from='client.15462 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-09T00:06:43.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:43 smithi102 ceph-mon[36106]: pgmap v372: 129 pgs: 129 active+clean; 645 KiB data, 388 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 767 B/s wr, 2 op/s 2024-03-09T00:06:43.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/980500950' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/129816780' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:43 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3420449304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:06:43.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.521+0000 7f3e105a9640 1 -- 172.21.15.73:0/2018376730 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08104720 msgr2=0x7f3e08106b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.521+0000 7f3e105a9640 1 --2- 172.21.15.73:0/2018376730 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08104720 0x7f3e08106b10 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f3df8009900 tx=0x7f3df802bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:43.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- 172.21.15.73:0/2018376730 shutdown_connections 2024-03-09T00:06:43.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- 172.21.15.73:0/2018376730 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08107050 0x7f3e08109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- 172.21.15.73:0/2018376730 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08104720 0x7f3e08106b10 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- 172.21.15.73:0/2018376730 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08101df0 0x7f3e081041e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- 172.21.15.73:0/2018376730 >> 172.21.15.73:0/2018376730 conn(0x7f3e080fb780 msgr2=0x7f3e080fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- 172.21.15.73:0/2018376730 shutdown_connections 2024-03-09T00:06:43.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- 172.21.15.73:0/2018376730 wait complete. 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 Processor -- start 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- start start 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08101df0 0x7f3e081975a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08104720 0x7f3e08197ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0810cf60 con 0x7f3e08107050 2024-03-09T00:06:43.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0810cde0 con 0x7f3e08104720 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e105a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e0810d0e0 con 0x7f3e08101df0 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.522+0000 7f3e0eb46640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60370/0 (socket says 172.21.15.73:60370) 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 -- 172.21.15.73:0/1823161340 learned_addr learned my addr 172.21.15.73:0/1823161340 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0e345640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08101df0 0x7f3e081975a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0db44640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08104720 0x7f3e08197ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 -- 172.21.15.73:0/1823161340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08101df0 msgr2=0x7f3e081975a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08101df0 0x7f3e081975a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 -- 172.21.15.73:0/1823161340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08104720 msgr2=0x7f3e08197ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08104720 0x7f3e08197ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 -- 172.21.15.73:0/1823161340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e0819f1c0 con 0x7f3e08107050 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e0eb46640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400eaa0 tx=0x7f3e0400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e0400cdb0 con 0x7f3e08107050 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e0819f4b0 con 0x7f3e08107050 2024-03-09T00:06:43.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e04005020 con 0x7f3e08107050 2024-03-09T00:06:43.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e040116b0 con 0x7f3e08107050 2024-03-09T00:06:43.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.523+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e0819f9d0 con 0x7f3e08107050 2024-03-09T00:06:43.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.524+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e080fd270 con 0x7f3e08107050 2024-03-09T00:06:43.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 -- 172.21.15.73:0/1498866387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f00101300 msgr2=0x7f5f00101720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/1498866387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f00101300 0x7f5f00101720 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f5efc009900 tx=0x7f5efc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 -- 172.21.15.73:0/1498866387 shutdown_connections 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/1498866387 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00109740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/1498866387 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f001024f0 0x7f5f00102970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/1498866387 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f00101300 0x7f5f00101720 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 -- 172.21.15.73:0/1498866387 >> 172.21.15.73:0/1498866387 conn(0x7f5f000fc890 msgr2=0x7f5f000fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 -- 172.21.15.73:0/1498866387 shutdown_connections 2024-03-09T00:06:43.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.525+0000 7f5f06b7e640 1 -- 172.21.15.73:0/1498866387 wait complete. 2024-03-09T00:06:43.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f3e04005560 con 0x7f3e08107050 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 Processor -- start 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 -- start start 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f00101300 0x7f5f000750a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00073c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0010cf90 con 0x7f5f001024f0 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0010ce10 con 0x7f5f00102eb0 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f06b7e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f0010d110 con 0x7f5f00101300 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60388/0 (socket says 172.21.15.73:60388) 2024-03-09T00:06:43.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 -- 172.21.15.73:0/559292255 learned_addr learned my addr 172.21.15.73:0/559292255 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f3dff7fe640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 0x7f3df40a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f0491a640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f00101300 0x7f5f000750a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f0511b640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00073c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f3e0e345640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 0x7f3df40a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3e040cf3d0 con 0x7f3e08107050 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 -- 172.21.15.73:0/559292255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f00101300 msgr2=0x7f5f000750a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f00101300 0x7f5f000750a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 -- 172.21.15.73:0/559292255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 msgr2=0x7f5f00073c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00073c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5ef7fff640 1 -- 172.21.15.73:0/559292255 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f000744b0 con 0x7f5f001024f0 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.526+0000 7f5f0511b640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.527+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000e9f0 tx=0x7f5ef000edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.527+0000 7f3e0e345640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 0x7f3df40a4d60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000bbe0 tx=0x7f3e0000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f5ef5ffb640 1 -- 172.21.15.73:0/559292255 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef000cd80 con 0x7f5f001024f0 2024-03-09T00:06:43.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f5ef5ffb640 1 -- 172.21.15.73:0/559292255 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef0005500 con 0x7f5f001024f0 2024-03-09T00:06:43.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f001a17c0 con 0x7f5f001024f0 2024-03-09T00:06:43.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f5ef5ffb640 1 -- 172.21.15.73:0/559292255 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef0010690 con 0x7f5f001024f0 2024-03-09T00:06:43.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3e04016030 con 0x7f3e08107050 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.530+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f001a1d10 con 0x7f5f001024f0 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.532+0000 7f5ef5ffb640 1 -- 172.21.15.73:0/559292255 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7f5ef00046c0 con 0x7f5f001024f0 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 Processor -- start 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5ef5ffb640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 0x7f5ee80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 -- start start 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f101c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f15290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0511b640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 0x7f5ee80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5e74b40 con 0x5629b5f15900 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5e1dd80 con 0x5629b5f0fda0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f09764740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5e70be0 con 0x5629b5f10790 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5ef5ffb640 1 -- 172.21.15.73:0/559292255 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5ef00cf3a0 con 0x7f5f001024f0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0511b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0511b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60398/0 (socket says 172.21.15.73:60398) 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0511b640 1 -- 172.21.15.73:0/1983569755 learned_addr learned my addr 172.21.15.73:0/1983569755 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f15290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0491a640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f101c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.533+0000 7f5f0511b640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 0x7f5ee80a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80138c0 tx=0x7f5ef8013cc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 -- 172.21.15.73:0/1983569755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 msgr2=0x5629b5f15290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f15290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 -- 172.21.15.73:0/1983569755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 msgr2=0x5629b5f101c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f101c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 -- 172.21.15.73:0/1983569755 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629b5c517e0 con 0x5629b5f15900 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0491a640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5f0511b640 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8011c60 tx=0x7f5ef801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71614fab91e7869e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5ee6ffd640 1 -- 172.21.15.73:0/1983569755 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef800de60 con 0x5629b5f15900 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5ee6ffd640 1 -- 172.21.15.73:0/1983569755 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef8011c90 con 0x5629b5f15900 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.534+0000 7f5ee6ffd640 1 -- 172.21.15.73:0/1983569755 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef801e630 con 0x5629b5f15900 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- 172.21.15.73:0/1983569755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 msgr2=0x5629b5f17d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f5ef8011c60 tx=0x7f5ef801d000 comp rx=0 tx=0).stop 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- 172.21.15.73:0/1983569755 shutdown_connections 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f15900 0x5629b5f17d90 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f15290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 --2- 172.21.15.73:0/1983569755 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f101c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- 172.21.15.73:0/1983569755 >> 172.21.15.73:0/1983569755 conn(0x5629b5f0b310 msgr2=0x5629b5f0b710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- 172.21.15.73:0/1983569755 shutdown_connections 2024-03-09T00:06:43.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- 172.21.15.73:0/1983569755 wait complete. 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 -- start start 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.535+0000 7f5f09764740 1 Processor -- start 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f10790 0x5629b5f1a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f21510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5f18850 con 0x5629b5f0fda0 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5e311c0 con 0x5629b5f10790 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629b5f189d0 con 0x5629b5f15900 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60410/0 (socket says 172.21.15.73:60410) 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 -- 172.21.15.73:0/573474559 learned_addr learned my addr 172.21.15.73:0/573474559 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f10790 0x5629b5f1a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 10 client.0 _ll_register_callbacks cb 0x5629b5e207a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:43.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0511b640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f21510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 -- 172.21.15.73:0/573474559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 msgr2=0x5629b5f21510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f21510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 -- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f10790 msgr2=0x5629b5f1a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f10790 0x5629b5f1a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 -- 172.21.15.73:0/573474559 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629b5cb1f70 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f0491a640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5f00102350 tx=0x7f5efc004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9c501a66acacc2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5edffff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc005d50 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5efc005230 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.536+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5ef8011c90 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5efc005430 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 10 client.15522 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5629b5b49170 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5ef8011c90 con 0x5629b5f0fda0 2024-03-09T00:06:43.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5efc003a10 con 0x5629b5f0fda0 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 10 client.15522 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 10 client.15522 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 20 client.15522 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f25d90 0x5629b5f281e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 4 client.15522 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5629b5e4a960 con 0x5629b5f25d90 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.537+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f25d90 0x5629b5f281e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f25d90 0x5629b5f281e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f80a8e4099dbf0b5 server_cookie=a3ebfb2023baac95 in_seq=0 out_seq=0 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5edffff640 10 client.15522 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5edffff640 1 -- 172.21.15.73:0/573474559 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5629b5e4a960 con 0x5629b5f25d90 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5edffff640 10 client.15522 handle_command_reply: tid=0 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 client.15522 shutdown 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 msgr2=0x5629b5f19f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.540 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f5f00102350 tx=0x7f5efc004270 comp rx=0 tx=0).stop 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 shutdown_connections 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f25d90 0x5629b5f281e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f21510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f10790 0x5629b5f1a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.538+0000 7f5f09764740 1 --2- 172.21.15.73:0/573474559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f19f90 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.539+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 >> 172.21.15.73:0/573474559 conn(0x5629b5f0b310 msgr2=0x5629b5f0a480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.539+0000 7f5edffff640 0 client.15522 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.539+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 shutdown_connections 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.539+0000 7f5f09764740 1 -- 172.21.15.73:0/573474559 wait complete. 2024-03-09T00:06:43.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.539+0000 7f5f09764740 20 client.15522 trim_cache size 0 max 16384 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.553+0000 7f5f09764740 1 Processor -- start 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 -- start start 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f214d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f15900 0x5629b5f0a130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001aa160 con 0x5629b5f0fda0 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001ac720 con 0x5629b5f15900 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f09764740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001aa760 con 0x5629b5f10790 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60412/0 (socket says 172.21.15.73:60412) 2024-03-09T00:06:43.553 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 -- 172.21.15.73:0/837986055 learned_addr learned my addr 172.21.15.73:0/837986055 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0511b640 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f15900 0x5629b5f0a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f214d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 -- 172.21.15.73:0/837986055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 msgr2=0x5629b5f214d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f214d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 -- 172.21.15.73:0/837986055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f15900 msgr2=0x5629b5f0a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f15900 0x5629b5f0a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 -- 172.21.15.73:0/837986055 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629b5f10b70 con 0x5629b5f0fda0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5f0491a640 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5efc02c7b0 tx=0x7f5efc009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bcc38e5ca27ebb7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5ee67fc640 1 -- 172.21.15.73:0/837986055 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc005d50 con 0x5629b5f0fda0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5ee67fc640 1 -- 172.21.15.73:0/837986055 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5efc009520 con 0x5629b5f0fda0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.554+0000 7f5ee67fc640 1 -- 172.21.15.73:0/837986055 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5efc034a70 con 0x5629b5f0fda0 2024-03-09T00:06:43.554 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- 172.21.15.73:0/837986055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 msgr2=0x5629b5f09bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f5efc02c7b0 tx=0x7f5efc009780 comp rx=0 tx=0).stop 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- 172.21.15.73:0/837986055 shutdown_connections 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f15900 0x5629b5f0a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f10790 0x5629b5f214d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 --2- 172.21.15.73:0/837986055 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f0fda0 0x5629b5f09bf0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- 172.21.15.73:0/837986055 >> 172.21.15.73:0/837986055 conn(0x5629b5f0b310 msgr2=0x5629b5f119d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- 172.21.15.73:0/837986055 shutdown_connections 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- 172.21.15.73:0/837986055 wait complete. 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 -- start start 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.555+0000 7f5f09764740 1 Processor -- start 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f21320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f1fe70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001aada0 con 0x5629b5f10790 2024-03-09T00:06:43.555 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001ac0c0 con 0x5629b5f0fda0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f001ab400 con 0x5629b5f15900 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f09764740 10 client.0 _ll_register_callbacks cb 0x7f5ef8013f10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60426/0 (socket says 172.21.15.73:60426) 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 -- 172.21.15.73:0/3231415979 learned_addr learned my addr 172.21.15.73:0/3231415979 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f0511b640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f1fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5f0491a640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f21320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 -- 172.21.15.73:0/3231415979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 msgr2=0x5629b5f1fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f1fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 -- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 msgr2=0x5629b5f21320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f21320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 -- 172.21.15.73:0/3231415979 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5629b5f205b0 con 0x5629b5f10790 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5ef7fff640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0097d90 tx=0x7f5ef00cf040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=795a5a40504edcde server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.556+0000 7f5edffff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5ef00cf0b0 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5ef0016db0 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5efc009520 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5ef00ba120 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 10 client.15528 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5629b5f7d310 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5efc009520 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5ef00860b0 con 0x5629b5f10790 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 10 client.15528 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 10 client.15528 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 20 client.15528 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f7dc70 0x5629b5f80090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 4 client.15528 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:06:43.557 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5629b5e36080 con 0x5629b5f7dc70 2024-03-09T00:06:43.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.557+0000 7f5f0491a640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f7dc70 0x5629b5f80090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.558+0000 7f5f0491a640 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f7dc70 0x5629b5f80090 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=59a5de70cac78238 server_cookie=425c4699e6ce3f89 in_seq=0 out_seq=0 2024-03-09T00:06:43.558 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.558+0000 7f5edffff640 10 client.15528 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5edffff640 1 -- 172.21.15.73:0/3231415979 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5629b5e36080 con 0x5629b5f7dc70 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5edffff640 10 client.15528 handle_command_reply: tid=0 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 client.15528 shutdown 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 msgr2=0x5629b5f1f930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f5ef0097d90 tx=0x7f5ef00cf040 comp rx=0 tx=0).stop 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 shutdown_connections 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5629b5f7dc70 0x5629b5f80090 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5629b5f15900 0x5629b5f1fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5629b5f10790 0x5629b5f1f930 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.559 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 --2- 172.21.15.73:0/3231415979 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5629b5f0fda0 0x5629b5f21320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5edffff640 0 client.15528 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.559+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 >> 172.21.15.73:0/3231415979 conn(0x5629b5f0b310 msgr2=0x5629b5f0e570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 shutdown_connections 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f09764740 1 -- 172.21.15.73:0/3231415979 wait complete. 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f09764740 20 client.15528 trim_cache size 0 max 16384 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "4319a4b7-33dc-4811-8f5b-fad980a06c0f", 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:06:43.560 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:06:43.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 msgr2=0x7f5ee80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 0x7f5ee80a4d40 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f5ef80138c0 tx=0x7f5ef8013cc0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.561 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 msgr2=0x7f5f000736e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5ef000e9f0 tx=0x7f5ef000edf0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 shutdown_connections 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f5ee80a2880 0x7f5ee80a4d40 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5f00102eb0 0x7f5f00073c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5f001024f0 0x7f5f000736e0 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 --2- 172.21.15.73:0/559292255 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5f00101300 0x7f5f000750a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 >> 172.21.15.73:0/559292255 conn(0x7f5f000fc890 msgr2=0x7f5f001058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 shutdown_connections 2024-03-09T00:06:43.562 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.560+0000 7f5f06b7e640 1 -- 172.21.15.73:0/559292255 wait complete. 2024-03-09T00:06:43.566 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:06:43.566 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:06:43.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 -- 172.21.15.73:0/1786436970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc1001f0 msgr2=0x7fa2bc100610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1786436970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc100610 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac009900 tx=0x7fa2ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 -- 172.21.15.73:0/1786436970 shutdown_connections 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1786436970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc101da0 0x7fa2bc108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1786436970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1013e0 0x7fa2bc101860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1786436970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc100610 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 -- 172.21.15.73:0/1786436970 >> 172.21.15.73:0/1786436970 conn(0x7fa2bc0fb780 msgr2=0x7fa2bc0fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 -- 172.21.15.73:0/1786436970 shutdown_connections 2024-03-09T00:06:43.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.676+0000 7fa2c084b640 1 -- 172.21.15.73:0/1786436970 wait complete. 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 Processor -- start 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 -- start start 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc1976e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc1013e0 0x7fa2bc197c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10be80 con 0x7fa2bc101da0 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10bd00 con 0x7fa2bc1001f0 2024-03-09T00:06:43.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2c084b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc10c000 con 0x7fa2bc1013e0 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60456/0 (socket says 172.21.15.73:60456) 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 -- 172.21.15.73:0/1020268878 learned_addr learned my addr 172.21.15.73:0/1020268878 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc1976e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc1013e0 0x7fa2bc197c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 -- 172.21.15.73:0/1020268878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc1013e0 msgr2=0x7fa2bc197c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc1013e0 0x7fa2bc197c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 -- 172.21.15.73:0/1020268878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1001f0 msgr2=0x7fa2bc1976e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc1976e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.677+0000 7fa2bad9d640 1 -- 172.21.15.73:0/1020268878 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2bc19f2b0 con 0x7fa2bc101da0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2bad9d640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400eb10 tx=0x7fa2b400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2ab7fe640 1 -- 172.21.15.73:0/1020268878 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4004b20 con 0x7fa2bc101da0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2ab7fe640 1 -- 172.21.15.73:0/1020268878 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b4004cc0 con 0x7fa2bc101da0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2bc19f5a0 con 0x7fa2bc101da0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2ab7fe640 1 -- 172.21.15.73:0/1020268878 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b4010760 con 0x7fa2bc101da0 2024-03-09T00:06:43.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.678+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2bc19faf0 con 0x7fa2bc101da0 2024-03-09T00:06:43.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2ab7fe640 1 -- 172.21.15.73:0/1020268878 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 18) v1 ==== 137290+0+0 (secure 0 0 0) 0x7fa2b4005020 con 0x7fa2bc101da0 2024-03-09T00:06:43.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 Processor -- start 2024-03-09T00:06:43.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 -- start start 2024-03-09T00:06:43.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ecdc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0e2ce00 con 0x560ef0ec8780 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0dd5b10 con 0x560ef0ece2e0 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2c342b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0e28ea0 con 0x560ef0ec9170 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60460/0 (socket says 172.21.15.73:60460) 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/1599735925 learned_addr learned my addr 172.21.15.73:0/1599735925 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2ab7fe640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 0x7fa2a40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.681+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ecdc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/1599735925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 msgr2=0x560ef0ecdc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2bad9d640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 0x7fa2a40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ecdc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/1599735925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 msgr2=0x560ef0ed0770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/1599735925 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef0c0b7d0 con 0x560ef0ec8780 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2ab7fe640 1 -- 172.21.15.73:0/1020268878 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa2b40cf4f0 con 0x7fa2bc101da0 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d40 tx=0x7fa2b0013830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7c17ea741c16ea9 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2a8ff9640 1 -- 172.21.15.73:0/1599735925 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b001c470 con 0x560ef0ec8780 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2a8ff9640 1 -- 172.21.15.73:0/1599735925 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b0017070 con 0x560ef0ec8780 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2a8ff9640 1 -- 172.21.15.73:0/1599735925 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b000f6d0 con 0x560ef0ec8780 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.682+0000 7fa2bad9d640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 0x7fa2a40a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40984b0 tx=0x7fa2b40668a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- 172.21.15.73:0/1599735925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 msgr2=0x560ef0ec8ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009d40 tx=0x7fa2b0013830 comp rx=0 tx=0).stop 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- 172.21.15.73:0/1599735925 shutdown_connections 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed0770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ecdc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1599735925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec8780 0x560ef0ec8ba0 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- 172.21.15.73:0/1599735925 >> 172.21.15.73:0/1599735925 conn(0x560ef0ec3cf0 msgr2=0x560ef0ec40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- 172.21.15.73:0/1599735925 shutdown_connections 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- 172.21.15.73:0/1599735925 wait complete. 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- start start 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 Processor -- start 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ed2750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0ed1290 con 0x560ef0ec9170 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0de90a0 con 0x560ef0ece2e0 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560ef0ed1410 con 0x560ef0ec8780 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.683+0000 7fa2c342b740 10 client.0 _ll_register_callbacks cb 0x560ef0de8a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60468/0 (socket says 172.21.15.73:60468) 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 -- 172.21.15.73:0/1128253266 learned_addr learned my addr 172.21.15.73:0/1128253266 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 -- 172.21.15.73:0/1128253266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 msgr2=0x560ef0ed2750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ed2750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 -- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 msgr2=0x560ef0ed9cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed9cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 -- 172.21.15.73:0/1128253266 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ef0b413f0 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac02c7b0 tx=0x7fa2ac004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=33b60b359a5c564e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2ac005d50 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa2ac005230 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2b0017070 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2ac005430 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2c342b740 10 client.15540 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560ef0b010f0 con 0x560ef0ec9170 2024-03-09T00:06:43.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.684+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b0017070 con 0x560ef0ec9170 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa2ac0037b0 con 0x560ef0ec9170 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 10 client.15540 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 10 client.15540 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 20 client.15540 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0ee0710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 4 client.15540 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560ef0e3a340 con 0x560ef0ede2c0 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0ee0710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2a1ffb640 10 client.15540 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.685+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0ee0710 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b803b32a95c353ba server_cookie=faf542468bd30f6f in_seq=0 out_seq=0 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/1128253266 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560ef0e3a340 con 0x560ef0ede2c0 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2a1ffb640 10 client.15540 handle_command_reply: tid=0 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 client.15540 shutdown 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 msgr2=0x560ef0ed2c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 crypto rx=0x7fa2ac02c7b0 tx=0x7fa2ac004270 comp rx=0 tx=0).stop 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 shutdown_connections 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0ee0710 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ece2e0 0x560ef0ed9cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ec9170 0x560ef0ed2c90 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2a1ffb640 0 client.15540 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 --2- 172.21.15.73:0/1128253266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ed2750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 >> 172.21.15.73:0/1128253266 conn(0x560ef0ec3cf0 msgr2=0x560ef0ec2ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 shutdown_connections 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 1 -- 172.21.15.73:0/1128253266 wait complete. 2024-03-09T00:06:43.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.686+0000 7fa2c342b740 20 client.15540 trim_cache size 0 max 16384 2024-03-09T00:06:43.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 Processor -- start 2024-03-09T00:06:43.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 -- start start 2024-03-09T00:06:43.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ec2660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec9170 0x560ef0ed9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a8000 con 0x560ef0ece2e0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1aa5c0 con 0x560ef0ec9170 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2c342b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a8600 con 0x560ef0ec8780 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60484/0 (socket says 172.21.15.73:60484) 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/2233160946 learned_addr learned my addr 172.21.15.73:0/2233160946 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/2233160946 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 msgr2=0x560ef0ec2660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ec2660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/2233160946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec9170 msgr2=0x560ef0ed9c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec9170 0x560ef0ed9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/2233160946 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560ef0ebf4d0 con 0x560ef0ece2e0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.701+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009250 tx=0x7fa2b00133a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9adb3f33f4ba4b6b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2a3fff640 1 -- 172.21.15.73:0/2233160946 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b002e070 con 0x560ef0ece2e0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2a3fff640 1 -- 172.21.15.73:0/2233160946 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b0017040 con 0x560ef0ece2e0 2024-03-09T00:06:43.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2a3fff640 1 -- 172.21.15.73:0/2233160946 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b00292b0 con 0x560ef0ece2e0 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- 172.21.15.73:0/2233160946 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 msgr2=0x560ef0ec2ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0009250 tx=0x7fa2b00133a0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- 172.21.15.73:0/2233160946 shutdown_connections 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ec2ba0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec9170 0x560ef0ed9c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 --2- 172.21.15.73:0/2233160946 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec8780 0x560ef0ec2660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- 172.21.15.73:0/2233160946 >> 172.21.15.73:0/2233160946 conn(0x560ef0ec3cf0 msgr2=0x560ef0ec77e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- 172.21.15.73:0/2233160946 shutdown_connections 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- 172.21.15.73:0/2233160946 wait complete. 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.702+0000 7fa2c342b740 1 -- start start 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 Processor -- start 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec8780 0x560ef0ed8380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ed88c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a8c40 con 0x560ef0ece2e0 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a9f60 con 0x560ef0ec8780 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2bc1a92a0 con 0x560ef0ec9170 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 10 client.0 _ll_register_callbacks cb 0x7fa2b40093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60490/0 (socket says 172.21.15.73:60490) 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/4092369076 learned_addr learned my addr 172.21.15.73:0/4092369076 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec8780 0x560ef0ed8380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ed88c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/4092369076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 msgr2=0x560ef0ed88c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ed88c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec8780 msgr2=0x560ef0ed8380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec8780 0x560ef0ed8380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 -- 172.21.15.73:0/4092369076 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560ef0f347e0 con 0x560ef0ece2e0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2b9d9b640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0003fd0 tx=0x7fa2b0012040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a95ec16d7261ad13 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa2b002e070 con 0x560ef0ece2e0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7fa2b0005380 con 0x560ef0ece2e0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.703+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa2b0017040 con 0x560ef0ece2e0 2024-03-09T00:06:43.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa2b00035f0 con 0x560ef0ece2e0 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 10 client.15546 fetch_fsmap learned FSMap version 20 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560ef0f359f0 con 0x560ef0ece2e0 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa2b0017040 con 0x560ef0ece2e0 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa2b001caa0 con 0x560ef0ece2e0 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 10 client.15546 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 10 client.15546 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 20 client.15546 populate_metadata read hostname 'smithi073' 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0f38010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 4 client.15546 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560ef0e34690 con 0x560ef0ede2c0 2024-03-09T00:06:43.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.704+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0f38010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2a1ffb640 10 client.15546 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2ba59c640 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0f38010 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9f70f2f062f9ba1c server_cookie=a46451a95acbcbaa in_seq=0 out_seq=0 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2a1ffb640 1 -- 172.21.15.73:0/4092369076 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560ef0e34690 con 0x560ef0ede2c0 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2a1ffb640 10 client.15546 handle_command_reply: tid=0 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 client.15546 shutdown 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 msgr2=0x560ef0ed8e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 crypto rx=0x7fa2b0003fd0 tx=0x7fa2b0012040 comp rx=0 tx=0).stop 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 shutdown_connections 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560ef0ede2c0 0x560ef0f38010 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560ef0ece2e0 0x560ef0ed8e00 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560ef0ec9170 0x560ef0ed88c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2a1ffb640 0 client.15546 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:06:43.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 --2- 172.21.15.73:0/4092369076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560ef0ec8780 0x560ef0ed8380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 >> 172.21.15.73:0/4092369076 conn(0x560ef0ec3cf0 msgr2=0x560ef0ec9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 shutdown_connections 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 1 -- 172.21.15.73:0/4092369076 wait complete. 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c342b740 20 client.15546 trim_cache size 0 max 16384 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "4319a4b7-33dc-4811-8f5b-fad980a06c0f": { 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "4319a4b7-33dc-4811-8f5b-fad980a06c0f", 2024-03-09T00:06:43.707 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 msgr2=0x7fa2a40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 0x7fa2a40a4cc0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fa2b40984b0 tx=0x7fa2b40668a0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 msgr2=0x7fa2bc19ec40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.705+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fa2b400eb10 tx=0x7fa2b400c2d0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 shutdown_connections 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7fa2a40a2800 0x7fa2a40a4cc0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa2bc101da0 0x7fa2bc19ec40 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa2bc1013e0 0x7fa2bc197c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 --2- 172.21.15.73:0/1020268878 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa2bc1001f0 0x7fa2bc1976e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 >> 172.21.15.73:0/1020268878 conn(0x7fa2bc0fb780 msgr2=0x7fa2bc0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 shutdown_connections 2024-03-09T00:06:43.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.706+0000 7fa2c084b640 1 -- 172.21.15.73:0/1020268878 wait complete. 2024-03-09T00:06:43.711 INFO:tasks.cephfs.filesystem:scrub status for tag:4319a4b7-33dc-4811-8f5b-fad980a06c0f - {'path': '/', 'tag': '4319a4b7-33dc-4811-8f5b-fad980a06c0f', 'options': 'recursive,force'} 2024-03-09T00:06:43.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:43.770+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "snap_schedule"} v 0) v1 -- 0x7f3e0802ce70 con 0x7f3e08107050 2024-03-09T00:06:44.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.048+0000 7f3dff7fe640 1 -- 172.21.15.73:0/1823161340 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "snap_schedule"}]=0 v19) v1 ==== 92+0+0 (secure 0 0 0) 0x7f3e0409ba70 con 0x7f3e08107050 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 msgr2=0x7f3df40a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 0x7f3df40a4d60 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f3e0000bbe0 tx=0x7f3e0000b390 comp rx=0 tx=0).stop 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 msgr2=0x7f3e0819eb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f3e0400eaa0 tx=0x7f3e0400ef60 comp rx=0 tx=0).stop 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 shutdown_connections 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3df40a28a0 0x7f3df40a4d60 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e08107050 0x7f3e0819eb50 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e08104720 0x7f3e08197ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 --2- 172.21.15.73:0/1823161340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e08101df0 0x7f3e081975a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 >> 172.21.15.73:0/1823161340 conn(0x7f3e080fb780 msgr2=0x7f3e08102850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.051+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 shutdown_connections 2024-03-09T00:06:44.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.052+0000 7f3e105a9640 1 -- 172.21.15.73:0/1823161340 wait complete. 2024-03-09T00:06:44.063 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/allow_m_granularity true' 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- 172.21.15.73:0/2671700780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101dc0 msgr2=0x7f27f4108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/2671700780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101dc0 0x7f27f4108650 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7f27ec009ca0 tx=0x7f27ec02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- 172.21.15.73:0/2671700780 shutdown_connections 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/2671700780 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101dc0 0x7f27f4108650 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/2671700780 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4101400 0x7f27f4101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/2671700780 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4100210 0x7f27f4100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- 172.21.15.73:0/2671700780 >> 172.21.15.73:0/2671700780 conn(0x7f27f40fb780 msgr2=0x7f27f40fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- 172.21.15.73:0/2671700780 shutdown_connections 2024-03-09T00:06:44.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- 172.21.15.73:0/2671700780 wait complete. 2024-03-09T00:06:44.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 Processor -- start 2024-03-09T00:06:44.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.176+0000 7f27fa6d1640 1 -- start start 2024-03-09T00:06:44.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4100210 0x7f27f4196410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4101dc0 0x7f27f419d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410bea0 con 0x7f27f4101400 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410bd20 con 0x7f27f4101dc0 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27fa6d1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27f410c020 con 0x7f27f4100210 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60532/0 (socket says 172.21.15.73:60532) 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 -- 172.21.15.73:0/3036124448 learned_addr learned my addr 172.21.15.73:0/3036124448 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f3fff640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4100210 0x7f27f4196410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f8c6e640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4101dc0 0x7f27f419d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 -- 172.21.15.73:0/3036124448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4100210 msgr2=0x7f27f4196410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4100210 0x7f27f4196410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 -- 172.21.15.73:0/3036124448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4101dc0 msgr2=0x7f27f419d9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4101dc0 0x7f27f419d9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 -- 172.21.15.73:0/3036124448 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f419e030 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f37fe640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f27e800d950 tx=0x7f27e800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.177+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e8017030 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.178+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27e8004020 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.178+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27f419e320 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.178+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27e80047a0 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.178+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27f419e870 con 0x7f27f4101400 2024-03-09T00:06:44.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.179+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27f4107530 con 0x7f27f4101400 2024-03-09T00:06:44.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.180+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f27e80041d0 con 0x7f27f4101400 2024-03-09T00:06:44.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.180+0000 7f27f17fa640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 0x7f27dc0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.180+0000 7f27f3fff640 1 -- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 msgr2=0x7f27dc0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:44.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.180+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f27e80ce710 con 0x7f27f4101400 2024-03-09T00:06:44.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.180+0000 7f27f3fff640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 0x7f27dc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-09T00:06:44.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.185+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f27e809b400 con 0x7f27f4101400 2024-03-09T00:06:44.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2924942682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3604130425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-09T00:06:44.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:06:44.334 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: ignoring --setuser ceph since I am not root 2024-03-09T00:06:44.334 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: ignoring --setgroup ceph since I am not root 2024-03-09T00:06:44.334 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-mgr[38444]: -- 172.21.15.179:0/1407511270 <== mon.0 v2:172.21.15.73:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e03cb885a0 con 0x55e03cb5cc00 2024-03-09T00:06:44.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.366+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}] v 0) v1 -- 0x7f27f4061ef0 con 0x7f27f4101400 2024-03-09T00:06:44.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.370+0000 7f27f17fa640 1 -- 172.21.15.73:0/3036124448 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/allow_m_granularity}]=0 v31)=0 v31) v1 ==== 138+0+0 (secure 0 0 0) 0x7f27e809ad00 con 0x7f27f4101400 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 msgr2=0x7f27dc0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 0x7f27dc0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 msgr2=0x7f27f4196950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 crypto rx=0x7f27e800d950 tx=0x7f27e800de10 comp rx=0 tx=0).stop 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 shutdown_connections 2024-03-09T00:06:44.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f27dc0a28f0 0x7f27dc0a4db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27f4101dc0 0x7f27f419d9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27f4101400 0x7f27f4196950 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 --2- 172.21.15.73:0/3036124448 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27f4100210 0x7f27f4196410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 >> 172.21.15.73:0/3036124448 conn(0x7f27f40fb780 msgr2=0x7f27f4104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 shutdown_connections 2024-03-09T00:06:44.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.372+0000 7f27fa6d1640 1 -- 172.21.15.73:0/3036124448 wait complete. 2024-03-09T00:06:44.377 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setuser ceph since I am not root 2024-03-09T00:06:44.377 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: ignoring --setgroup ceph since I am not root 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2924942682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3604130425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-09T00:06:44.378 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:06:44.379 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph config set mgr mgr/snap_schedule/dump_on_update true' 2024-03-09T00:06:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 -- 172.21.15.73:0/3675768644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04100210 msgr2=0x7f5a04100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 --2- 172.21.15.73:0/3675768644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04100210 0x7f5a04100630 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 crypto rx=0x7f5a00009900 tx=0x7f5a0002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 -- 172.21.15.73:0/3675768644 shutdown_connections 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 --2- 172.21.15.73:0/3675768644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04101dc0 0x7f5a04108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 --2- 172.21.15.73:0/3675768644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101400 0x7f5a04101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 --2- 172.21.15.73:0/3675768644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04100210 0x7f5a04100630 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 -- 172.21.15.73:0/3675768644 >> 172.21.15.73:0/3675768644 conn(0x7f5a040fb780 msgr2=0x7f5a040fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 -- 172.21.15.73:0/3675768644 shutdown_connections 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.489+0000 7f5a0c620640 1 -- 172.21.15.73:0/3675768644 wait complete. 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 Processor -- start 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 -- start start 2024-03-09T00:06:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 0x7f5a04197600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101dc0 0x7f5a0419eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0410bea0 con 0x7f5a04101400 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0410bd20 con 0x7f5a04100210 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0c620640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a0410c020 con 0x7f5a04101dc0 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a09bbb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a09bbb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60552/0 (socket says 172.21.15.73:60552) 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a09bbb640 1 -- 172.21.15.73:0/2785725296 learned_addr learned my addr 172.21.15.73:0/2785725296 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 0x7f5a04197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0abbd640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101dc0 0x7f5a0419eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101dc0 msgr2=0x7f5a0419eb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101dc0 0x7f5a0419eb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 msgr2=0x7f5a04197b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a0a3bc640 1 -- 172.21.15.73:0/2785725296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a0419f1d0 con 0x7f5a04100210 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.490+0000 7f5a09bbb640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f5a0a3bc640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 0x7f5a04197600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c170 tx=0x7f5a0002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00041070 con 0x7f5a04100210 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a0419f460 con 0x7f5a04100210 2024-03-09T00:06:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a00004450 con 0x7f5a04100210 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a00003e50 con 0x7f5a04100210 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.491+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a0419f920 con 0x7f5a04100210 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.492+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a0419fb80 con 0x7f5a04100210 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.493+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f5a00044070 con 0x7f5a04100210 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.493+0000 7f59fb7fe640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 0x7f59f00a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.493+0000 7f5a09bbb640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 msgr2=0x7f59f00a4f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.494+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f5a000f08c0 con 0x7f5a04100210 2024-03-09T00:06:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.494+0000 7f5a09bbb640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 0x7f59f00a4f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-09T00:06:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.498+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f5a000bcf60 con 0x7f5a04100210 2024-03-09T00:06:44.508 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: ignoring --setuser ceph since I am not root 2024-03-09T00:06:44.508 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: ignoring --setgroup ceph since I am not root 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mgr[37529]: -- 172.21.15.102:0/3970189862 <== mon.2 v2:172.21.15.102:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x564e8ec425a0 con 0x564e8ec16800 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2924942682' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3604130425' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:06:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:06:44.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "snap_schedule"}]: dispatch 2024-03-09T00:06:44.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:06:44.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.695+0000 7f5a09bbb640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 msgr2=0x7f59f00a4f50 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:44.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.695+0000 7f5a09bbb640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 0x7f59f00a4f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-09T00:06:44.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.696+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command([{prefix=config set, name=mgr/snap_schedule/dump_on_update}] v 0) v1 -- 0x7f5a0419ff80 con 0x7f5a04100210 2024-03-09T00:06:44.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.702+0000 7f59fb7fe640 1 -- 172.21.15.73:0/2785725296 <== mon.2 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/snap_schedule/dump_on_update}]=0 v32)=0 v32) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5a000c1000 con 0x7f5a04100210 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 msgr2=0x7f59f00a4f50 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 0x7f59f00a4f50 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 msgr2=0x7f5a04197600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 0x7f5a04197600 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f5a0002c170 tx=0x7f5a0002c650 comp rx=0 tx=0).stop 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 shutdown_connections 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f59f00a2a90 0x7f59f00a4f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f5a04101dc0 0x7f5a0419eb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5a04101400 0x7f5a04197b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 --2- 172.21.15.73:0/2785725296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5a04100210 0x7f5a04197600 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 >> 172.21.15.73:0/2785725296 conn(0x7f5a040fb780 msgr2=0x7f5a04068970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 shutdown_connections 2024-03-09T00:06:44.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.704+0000 7f5a0c620640 1 -- 172.21.15.73:0/2785725296 wait complete. 2024-03-09T00:06:44.710 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule add --fs=cephfs --path=/ --snap_schedule=1m' 2024-03-09T00:06:44.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.822+0000 7f3d52a6e640 1 -- 172.21.15.73:0/288690085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 msgr2=0x7f3d4c108650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.822+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/288690085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c108650 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 crypto rx=0x7f3d44009ca0 tx=0x7f3d4402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- 172.21.15.73:0/288690085 shutdown_connections 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/288690085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c108650 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/288690085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c101400 0x7f3d4c101880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/288690085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 0x7f3d4c100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- 172.21.15.73:0/288690085 >> 172.21.15.73:0/288690085 conn(0x7f3d4c0fb780 msgr2=0x7f3d4c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- 172.21.15.73:0/288690085 shutdown_connections 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- 172.21.15.73:0/288690085 wait complete. 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 Processor -- start 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- start start 2024-03-09T00:06:44.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 0x7f3d4c069540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c101400 0x7f3d4c19c230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c19e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c10bea0 con 0x7f3d4c101dc0 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c10bd20 con 0x7f3d4c100210 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d52a6e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d4c10c020 con 0x7f3d4c101400 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d5100b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c19e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d5080a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 0x7f3d4c069540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d5100b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c19e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:60572/0 (socket says 172.21.15.73:60572) 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.823+0000 7f3d5100b640 1 -- 172.21.15.73:0/1864160428 learned_addr learned my addr 172.21.15.73:0/1864160428 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:44.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d4bfff640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c101400 0x7f3d4c19c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c101400 msgr2=0x7f3d4c19c230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3d4c101400 0x7f3d4c19c230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 msgr2=0x7f3d4c069540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 0x7f3d4c069540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d4c19f160 con 0x7f3d4c101dc0 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3d4c100210 0x7f3d4c069540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d5100b640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c19e8a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3d4402c630 tx=0x7f3d44032ea0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44041020 con 0x7f3d4c101dc0 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d44034cd0 con 0x7f3d4c101dc0 2024-03-09T00:06:44.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d4c19f3f0 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d44039040 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.824+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d4c19f8e0 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.826+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d4c1074d0 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.826+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 19) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f3d44044020 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.827+0000 7f3d49ffb640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.827+0000 7f3d5080a640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.827+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 9179+0+0 (secure 0 0 0) 0x7f3d440f0d00 con 0x7f3d4c101dc0 2024-03-09T00:06:44.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.827+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-09T00:06:44.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:44.831+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3d4404b020 con 0x7f3d4c101dc0 2024-03-09T00:06:44.901 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:44.650+0000 7fbf95631200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-09T00:06:44.901 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:44 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:44.750+0000 7fbf95631200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-09T00:06:45.008 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:44.614+0000 7f05929e4200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-09T00:06:45.009 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:44 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:44.706+0000 7f05929e4200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-09T00:06:45.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:45.028+0000 7f3d5080a640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:45.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:45.028+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-09T00:06:45.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:45.047+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d4c061730 con 0x7f3d3c0a28f0 2024-03-09T00:06:45.044 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:44.617+0000 7f0502cd7200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-09T00:06:45.044 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:44.709+0000 7f0502cd7200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-09T00:06:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-09T00:06:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-mon[36637]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-09T00:06:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3036124448' entity='client.admin' 2024-03-09T00:06:45.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-mon[36637]: from='client.? ' entity='client.admin' 2024-03-09T00:06:45.337 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-09T00:06:45.337 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-mon[36106]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-09T00:06:45.337 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3036124448' entity='client.admin' 2024-03-09T00:06:45.337 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-mon[36106]: from='client.? ' entity='client.admin' 2024-03-09T00:06:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1823161340' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "snap_schedule"}]': finished 2024-03-09T00:06:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-mon[32309]: mgrmap e19: x(active, since 9m), standbys: z, y 2024-03-09T00:06:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3036124448' entity='client.admin' 2024-03-09T00:06:45.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-mon[32309]: from='client.? ' entity='client.admin' 2024-03-09T00:06:45.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:45.429+0000 7f3d5080a640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:45.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:45.429+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-09T00:06:45.663 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:45.342+0000 7f05929e4200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-09T00:06:45.664 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:45.434+0000 7f05929e4200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-09T00:06:45.664 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:45.574+0000 7f05929e4200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-09T00:06:45.664 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:45.668+0000 7f05929e4200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-09T00:06:45.666 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:45.344+0000 7f0502cd7200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-09T00:06:45.666 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:45.437+0000 7f0502cd7200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-09T00:06:45.666 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:45.578+0000 7f0502cd7200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-09T00:06:45.666 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:45.672+0000 7f0502cd7200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-09T00:06:45.735 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:45.408+0000 7fbf95631200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-09T00:06:45.736 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:45.502+0000 7fbf95631200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-09T00:06:45.736 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:45.644+0000 7fbf95631200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-09T00:06:46.008 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:45 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:45.752+0000 7f05929e4200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-09T00:06:46.016 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:45.755+0000 7f0502cd7200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-09T00:06:46.079 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:45.739+0000 7fbf95631200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-09T00:06:46.079 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:45.822+0000 7fbf95631200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-09T00:06:46.079 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:46 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:46.084+0000 7fbf95631200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-09T00:06:46.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:46.230+0000 7f3d5080a640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:46.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:46.230+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-09T00:06:46.300 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:46.018+0000 7f05929e4200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-09T00:06:46.300 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:46.126+0000 7f05929e4200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:46.300 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:46.211+0000 7f05929e4200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-09T00:06:46.300 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:46.305+0000 7f05929e4200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-09T00:06:46.317 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:46.022+0000 7f0502cd7200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-09T00:06:46.318 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:46.135+0000 7f0502cd7200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:46.318 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:46.224+0000 7f0502cd7200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-09T00:06:46.318 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:46.323+0000 7f0502cd7200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-09T00:06:46.366 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:46 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:46.192+0000 7fbf95631200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:46.366 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:46 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:46.278+0000 7fbf95631200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-09T00:06:46.581 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:46.509+0000 7f0502cd7200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-09T00:06:46.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:46 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:46.371+0000 7fbf95631200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-09T00:06:46.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:46 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:46.544+0000 7fbf95631200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-09T00:06:46.759 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:46.478+0000 7f05929e4200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-09T00:06:47.581 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:47.492+0000 7f0502cd7200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-09T00:06:47.758 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:47 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:47.430+0000 7f05929e4200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-09T00:06:47.759 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:47 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:47.559+0000 7f05929e4200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-09T00:06:47.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:47.831+0000 7f3d5080a640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.73:6800/2701889057 2024-03-09T00:06:47.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:47.831+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-09T00:06:47.900 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:47 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:47.494+0000 7fbf95631200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-09T00:06:47.901 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:47 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:47.622+0000 7fbf95631200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-09T00:06:47.949 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:47.622+0000 7f0502cd7200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-09T00:06:48.163 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:47 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:47.888+0000 7f05929e4200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-09T00:06:48.163 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:47 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:47.977+0000 7f05929e4200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-09T00:06:48.163 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:48.169+0000 7f05929e4200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:48.232 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:47.955+0000 7f0502cd7200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-09T00:06:48.233 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:48.044+0000 7f0502cd7200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-09T00:06:48.238 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:47 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:47.952+0000 7fbf95631200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-09T00:06:48.238 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:48.043+0000 7fbf95631200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-09T00:06:48.508 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:48.352+0000 7f05929e4200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-09T00:06:48.581 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:48.238+0000 7f0502cd7200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:48.581 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:48.421+0000 7f0502cd7200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-09T00:06:48.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:48.244+0000 7fbf95631200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:48.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:48.431+0000 7fbf95631200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-09T00:06:48.880 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:48.631+0000 7f05929e4200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:48.880 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:48.886+0000 7f05929e4200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:48.959 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:48.703+0000 7f0502cd7200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:48.976 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:48.721+0000 7fbf95631200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-09T00:06:49.259 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:49.119+0000 7f05929e4200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-09T00:06:49.331 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:48.963+0000 7f0502cd7200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:49.331 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:49.198+0000 7f0502cd7200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-09T00:06:49.401 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:48.981+0000 7fbf95631200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-09T00:06:49.401 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:49 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:49.221+0000 7fbf95631200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-09T00:06:50.258 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:49.909+0000 7f05929e4200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-09T00:06:50.331 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:49.994+0000 7f0502cd7200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-09T00:06:50.401 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:50.029+0000 7fbf95631200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-09T00:06:50.758 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:50.476+0000 7f05929e4200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-09T00:06:50.759 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:50.573+0000 7f05929e4200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-09T00:06:50.831 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:50.568+0000 7f0502cd7200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-09T00:06:50.831 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:50.665+0000 7f0502cd7200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-09T00:06:50.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:50.858+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mgrmap(e 20) v1 ==== 137307+0+0 (secure 0 0 0) 0x7f3d44003970 con 0x7f3d4c101dc0 2024-03-09T00:06:50.855 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:50.604+0000 7fbf95631200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-09T00:06:50.855 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:50.704+0000 7fbf95631200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-09T00:06:50.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:50.956+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 8 ==== mgrmap(e 21) v1 ==== 136975+0+0 (secure 0 0 0) 0x7f3d440b99f0 con 0x7f3d4c101dc0 2024-03-09T00:06:50.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:50.956+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 msgr2=0x7f3d3c0a4db0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-09T00:06:50.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:50.956+0000 7f3d49ffb640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2701889057,v1:172.21.15.73:6801/2701889057] conn(0x7f3d3c0a28f0 0x7f3d3c0a4db0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:51.098 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y[37525]: 2024-03-09T00:06:50.785+0000 7f05929e4200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-09T00:06:51.098 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: Standby manager daemon y restarted 2024-03-09T00:06:51.098 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: Standby manager daemon y started 2024-03-09T00:06:51.098 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-09T00:06:51.099 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:51.099 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-09T00:06:51.099 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:50 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:50.929+0000 7fbf95631200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: Standby manager daemon y restarted 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: Standby manager daemon y started 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-09T00:06:51.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:50 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: Standby manager daemon y restarted 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: Standby manager daemon y started 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-09T00:06:51.187 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.102:0/3404080571' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:51.188 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:06:50 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z[38405]: 2024-03-09T00:06:50.881+0000 7f0502cd7200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-09T00:06:51.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.189+0000 7fbef7895700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.189+0000 7fbef7895700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.189+0000 7fbef7895700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.651 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.189+0000 7fbef7895700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.189+0000 7fbef7895700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.190+0000 7fbef488f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.190+0000 7fbef488f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.190+0000 7fbef488f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.190+0000 7fbef488f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:06:51.190+0000 7fbef488f700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:06:51.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:51.958+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mon.0 v2:172.21.15.73:3300/0 9 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f3d440ee4a0 con 0x7f3d4c101dc0 2024-03-09T00:06:51.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:51.959+0000 7f3d49ffb640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3d3c034a40 0x7f3d3c034e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:51.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:51.959+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}) v1 -- 0x7f3d4407cd00 con 0x7f3d3c034a40 2024-03-09T00:06:51.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:51.960+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3d3c034a40 0x7f3d3c034e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:51.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:51.965+0000 7f3d5080a640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3d3c034a40 0x7f3d3c034e20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000a010 tx=0x7f3d40005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.026+0000 7f3d49ffb640 1 -- 172.21.15.73:0/1864160428 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f3d4407cd00 con 0x7f3d3c034a40 2024-03-09T00:06:52.021 INFO:teuthology.orchestra.run.smithi073.stdout:Schedule set for path / 2024-03-09T00:06:52.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3d3c034a40 msgr2=0x7f3d3c034e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3d3c034a40 0x7f3d3c034e20 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f3d4000a010 tx=0x7f3d40005f90 comp rx=0 tx=0).stop 2024-03-09T00:06:52.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 msgr2=0x7f3d4c19e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d52a6e640 1 --2- 172.21.15.73:0/1864160428 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3d4c101dc0 0x7f3d4c19e8a0 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 crypto rx=0x7f3d4402c630 tx=0x7f3d44032ea0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d5100b640 1 -- 172.21.15.73:0/1864160428 reap_dead start 2024-03-09T00:06:52.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.028+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 shutdown_connections 2024-03-09T00:06:52.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.029+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 >> 172.21.15.73:0/1864160428 conn(0x7f3d4c0fb780 msgr2=0x7f3d4c1046a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.029+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 shutdown_connections 2024-03-09T00:06:52.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.029+0000 7f3d52a6e640 1 -- 172.21.15.73:0/1864160428 wait complete. 2024-03-09T00:06:52.034 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule retention add --fs=cephfs --path=/ --retention-spec-or-period=6m3h' 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- 172.21.15.73:0/726225079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 msgr2=0x7f4ba8100680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 --2- 172.21.15.73:0/726225079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8100680 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4009900 tx=0x7f4ba402bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- 172.21.15.73:0/726225079 shutdown_connections 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 --2- 172.21.15.73:0/726225079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101e10 0x7f4ba81086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 --2- 172.21.15.73:0/726225079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101450 0x7f4ba81018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 --2- 172.21.15.73:0/726225079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8100680 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- 172.21.15.73:0/726225079 >> 172.21.15.73:0/726225079 conn(0x7f4ba80fb7d0 msgr2=0x7f4ba80fdc10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- 172.21.15.73:0/726225079 shutdown_connections 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- 172.21.15.73:0/726225079 wait complete. 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 Processor -- start 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.145+0000 7f4bb020b640 1 -- start start 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101450 0x7f4ba81028a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 0x7f4ba8102de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba810bef0 con 0x7f4ba8100260 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba810bd70 con 0x7f4ba8101e10 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ba810c070 con 0x7f4ba8101450 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37982/0 (socket says 172.21.15.73:37982) 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 -- 172.21.15.73:0/2106765804 learned_addr learned my addr 172.21.15.73:0/2106765804 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bad7a6640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101450 0x7f4ba81028a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bae7a8640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 0x7f4ba8102de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 -- 172.21.15.73:0/2106765804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101450 msgr2=0x7f4ba81028a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101450 0x7f4ba81028a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 -- 172.21.15.73:0/2106765804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 msgr2=0x7f4ba8102de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 0x7f4ba8102de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 -- 172.21.15.73:0/2106765804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ba8103670 con 0x7f4ba8100260 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bae7a8640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 0x7f4ba8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4badfa7640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005ab0 tx=0x7f4ba4005dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4041070 con 0x7f4ba8100260 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4ba40054e0 con 0x7f4ba8100260 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ba81a1650 con 0x7f4ba8100260 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4ba4034440 con 0x7f4ba8100260 2024-03-09T00:06:52.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.146+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ba81a1bf0 con 0x7f4ba8100260 2024-03-09T00:06:52.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.147+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ba81a1e50 con 0x7f4ba8100260 2024-03-09T00:06:52.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.148+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f4ba4044070 con 0x7f4ba8100260 2024-03-09T00:06:52.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.149+0000 7f4b9effd640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 0x7f4b940a4bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.149+0000 7f4bad7a6640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 0x7f4b940a4bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.149+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f4ba40f0890 con 0x7f4ba8100260 2024-03-09T00:06:52.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.149+0000 7f4bad7a6640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 0x7f4b940a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8103fd0 tx=0x7f4b9800a2d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.153+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4ba40bd770 con 0x7f4ba8100260 2024-03-09T00:06:52.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-09T00:06:52.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: Standby manager daemon z restarted 2024-03-09T00:06:52.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: Standby manager daemon z started 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: Active manager daemon x restarted 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: Activating manager daemon x 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: osdmap e94: 12 total, 12 up, 12 in 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: mgrmap e21: x(active, starting, since 0.0157784s), standbys: y, z 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:06:52.152 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-09T00:06:52.153 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:06:52.154 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: Manager daemon x is now available 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-09T00:06:52.155 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:51 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: Standby manager daemon z restarted 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: Standby manager daemon z started 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: Active manager daemon x restarted 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: Activating manager daemon x 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: osdmap e94: 12 total, 12 up, 12 in 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: mgrmap e21: x(active, starting, since 0.0157784s), standbys: y, z 2024-03-09T00:06:52.173 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:06:52.174 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:06:52.175 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: Manager daemon x is now available 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-09T00:06:52.176 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:51 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-09T00:06:52.205 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: mgrmap e20: x(active, since 9m), standbys: z, y 2024-03-09T00:06:52.205 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: Standby manager daemon z restarted 2024-03-09T00:06:52.205 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: Standby manager daemon z started 2024-03-09T00:06:52.205 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-09T00:06:52.205 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.? 172.21.15.179:0/2046305507' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: Active manager daemon x restarted 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: Activating manager daemon x 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: osdmap e94: 12 total, 12 up, 12 in 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: mgrmap e21: x(active, starting, since 0.0157784s), standbys: y, z 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-09T00:06:52.206 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-09T00:06:52.207 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: Manager daemon x is now available 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.208 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-09T00:06:52.209 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:51 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-09T00:06:52.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.359+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ba806a1e0 con 0x7f4b940a2730 2024-03-09T00:06:52.410 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.415+0000 7f4b9effd640 1 -- 172.21.15.73:0/2106765804 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f4ba806a1e0 con 0x7f4b940a2730 2024-03-09T00:06:52.410 INFO:teuthology.orchestra.run.smithi073.stdout:Retention added to path / 2024-03-09T00:06:52.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 msgr2=0x7f4b940a4bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 0x7f4b940a4bf0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f4ba8103fd0 tx=0x7f4b9800a2d0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.412 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 msgr2=0x7f4ba8104260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 crypto rx=0x7f4ba4005ab0 tx=0x7f4ba4005dc0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 shutdown_connections 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4ba8101e10 0x7f4ba8102de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4ba8101450 0x7f4ba81028a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f4b940a2730 0x7f4b940a4bf0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 --2- 172.21.15.73:0/2106765804 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4ba8100260 0x7f4ba8104260 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 >> 172.21.15.73:0/2106765804 conn(0x7f4ba80fb7d0 msgr2=0x7f4ba8069ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 shutdown_connections 2024-03-09T00:06:52.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.417+0000 7f4bb020b640 1 -- 172.21.15.73:0/2106765804 wait complete. 2024-03-09T00:06:52.423 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-09T00:06:52.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 -- 172.21.15.73:0/1228224423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c101d90 msgr2=0x7f299c104180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 --2- 172.21.15.73:0/1228224423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c101d90 0x7f299c104180 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 crypto rx=0x7f298c009900 tx=0x7f298c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:52.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 -- 172.21.15.73:0/1228224423 shutdown_connections 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 --2- 172.21.15.73:0/1228224423 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c106ff0 0x7f299c109440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 --2- 172.21.15.73:0/1228224423 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c1046c0 0x7f299c106ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 --2- 172.21.15.73:0/1228224423 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c101d90 0x7f299c104180 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 -- 172.21.15.73:0/1228224423 >> 172.21.15.73:0/1228224423 conn(0x7f299c0fb780 msgr2=0x7f299c0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 -- 172.21.15.73:0/1228224423 shutdown_connections 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.533+0000 7f29a1e85640 1 -- 172.21.15.73:0/1228224423 wait complete. 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 Processor -- start 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 -- start start 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c101d90 0x7f299c101700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c1046c0 0x7f299c0ffd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c10cfa0 con 0x7f299c106ff0 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c10ce20 con 0x7f299c1046c0 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f29a1e85640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c10d120 con 0x7f299c101d90 2024-03-09T00:06:52.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f299bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f299bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38002/0 (socket says 172.21.15.73:38002) 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f299bfff640 1 -- 172.21.15.73:0/719604265 learned_addr learned my addr 172.21.15.73:0/719604265 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f299affd640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c1046c0 0x7f299c0ffd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.534+0000 7f299b7fe640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c101d90 0x7f299c101700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 -- 172.21.15.73:0/719604265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c101d90 msgr2=0x7f299c101700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c101d90 0x7f299c101700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 -- 172.21.15.73:0/719604265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c1046c0 msgr2=0x7f299c0ffd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c1046c0 0x7f299c0ffd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 -- 172.21.15.73:0/719604265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299c100b10 con 0x7f299c106ff0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f299bfff640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f299400eaa0 tx=0x7f299400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f299400cdb0 con 0x7f299c106ff0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2994005020 con 0x7f299c106ff0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c1a1690 con 0x7f299c106ff0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29940116b0 con 0x7f299c106ff0 2024-03-09T00:06:52.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.535+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c1a1b90 con 0x7f299c106ff0 2024-03-09T00:06:52.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.536+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c0fd270 con 0x7f299c106ff0 2024-03-09T00:06:52.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.537+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7f2994005560 con 0x7f299c106ff0 2024-03-09T00:06:52.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.537+0000 7f2998ff9640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 0x7f29840a4b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.538+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f29940ced30 con 0x7f299c106ff0 2024-03-09T00:06:52.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.538+0000 7f299b7fe640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 0x7f29840a4b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.538+0000 7f299b7fe640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 0x7f29840a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f298c02c7b0 tx=0x7f298c005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.542+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f2994016030 con 0x7f299c106ff0 2024-03-09T00:06:52.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.744+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f299c061730 con 0x7f29840a26b0 2024-03-09T00:06:52.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.752+0000 7f2998ff9640 1 -- 172.21.15.73:0/719604265 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+298 (secure 0 0 0) 0x7f299c061730 con 0x7f29840a26b0 2024-03-09T00:06:52.747 INFO:teuthology.orchestra.run.smithi073.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-09T00:00:00", "created": "2024-03-09T00:06:51", "first": null, "last": null, "last_pruned": null, "created_count": 0, "pruned_count": 0, "active": true} 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 msgr2=0x7f29840a4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 0x7f29840a4b70 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f298c02c7b0 tx=0x7f298c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 msgr2=0x7f299c100280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 crypto rx=0x7f299400eaa0 tx=0x7f299400ef60 comp rx=0 tx=0).stop 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 shutdown_connections 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f29840a26b0 0x7f29840a4b70 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f299c106ff0 0x7f299c100280 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f299c1046c0 0x7f299c0ffd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 --2- 172.21.15.73:0/719604265 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f299c101d90 0x7f299c101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.754+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 >> 172.21.15.73:0/719604265 conn(0x7f299c0fb780 msgr2=0x7f299c1026d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.755+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 shutdown_connections 2024-03-09T00:06:52.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.755+0000 7f29a1e85640 1 -- 172.21.15.73:0/719604265 wait complete. 2024-03-09T00:06:52.760 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- 172.21.15.73:0/2381061563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc101400 msgr2=0x7fe9cc101880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/2381061563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc101400 0x7fe9cc101880 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 crypto rx=0x7fe9c0009900 tx=0x7fe9c002bc30 comp rx=0 tx=0).stop 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- 172.21.15.73:0/2381061563 shutdown_connections 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/2381061563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101dc0 0x7fe9cc108650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/2381061563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc101400 0x7fe9cc101880 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/2381061563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc100210 0x7fe9cc100630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- 172.21.15.73:0/2381061563 >> 172.21.15.73:0/2381061563 conn(0x7fe9cc0fb780 msgr2=0x7fe9cc0fdbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- 172.21.15.73:0/2381061563 shutdown_connections 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- 172.21.15.73:0/2381061563 wait complete. 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 Processor -- start 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 -- start start 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.883+0000 7fe9d14f1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9d14f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101400 0x7fe9cc1958f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9d14f1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc101dc0 0x7fe9cc19c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9d14f1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc10ad90 con 0x7fe9cc100210 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9d14f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc10ac10 con 0x7fe9cc101400 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9d14f1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9cc10af10 con 0x7fe9cc101dc0 2024-03-09T00:06:52.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38024/0 (socket says 172.21.15.73:38024) 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9ca7fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101400 0x7fe9cc1958f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 -- 172.21.15.73:0/1785006224 learned_addr learned my addr 172.21.15.73:0/1785006224 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9cb7fe640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc101dc0 0x7fe9cc19c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 -- 172.21.15.73:0/1785006224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc101dc0 msgr2=0x7fe9cc19c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc101dc0 0x7fe9cc19c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 -- 172.21.15.73:0/1785006224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101400 msgr2=0x7fe9cc1958f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101400 0x7fe9cc1958f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 -- 172.21.15.73:0/1785006224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9cc19cfd0 con 0x7fe9cc100210 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.884+0000 7fe9caffd640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00e9c0 tx=0x7fe9bc00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.885+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc00cd80 con 0x7fe9cc100210 2024-03-09T00:06:52.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.885+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe9bc0035f0 con 0x7fe9cc100210 2024-03-09T00:06:52.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.885+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9cc19d2c0 con 0x7fe9cc100210 2024-03-09T00:06:52.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.885+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe9bc010690 con 0x7fe9cc100210 2024-03-09T00:06:52.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.885+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9cc19d790 con 0x7fe9cc100210 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.886+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9cc107530 con 0x7fe9cc100210 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.887+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 22) v1 ==== 137102+0+0 (secure 0 0 0) 0x7fe9bc0037a0 con 0x7fe9cc100210 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.887+0000 7fe9b3fff640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 0x7fe9b40a6d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.888+0000 7fe9ca7fc640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 0x7fe9b40a6d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.888+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe9bc0cef40 con 0x7fe9cc100210 2024-03-09T00:06:52.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.888+0000 7fe9ca7fc640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 0x7fe9b40a6d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9c002c7b0 tx=0x7fe9c0005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:06:52.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:52.891+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fe9bc09be20 con 0x7fe9cc100210 2024-03-09T00:06:53.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.114+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fe9cc061730 con 0x7fe9b40a48c0 2024-03-09T00:06:53.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.121+0000 7fe9b3fff640 1 -- 172.21.15.73:0/1785006224 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7fe9cc061730 con 0x7fe9b40a48c0 2024-03-09T00:06:53.115 INFO:teuthology.orchestra.run.smithi073.stdout:/ 1m 6m3h 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 msgr2=0x7fe9b40a6d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 0x7fe9b40a6d80 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fe9c002c7b0 tx=0x7fe9c0005fd0 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 msgr2=0x7fe9cc1953b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 crypto rx=0x7fe9bc00e9c0 tx=0x7fe9bc00ee80 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 shutdown_connections 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe9cc101dc0 0x7fe9cc19c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe9cc101400 0x7fe9cc1958f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe9b40a48c0 0x7fe9b40a6d80 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 --2- 172.21.15.73:0/1785006224 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe9cc100210 0x7fe9cc1953b0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:06:53.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 >> 172.21.15.73:0/1785006224 conn(0x7fe9cc0fb780 msgr2=0x7fe9cc104700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:06:53.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 shutdown_connections 2024-03-09T00:06:53.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:06:53.123+0000 7fe9d14f1640 1 -- 172.21.15.73:0/1785006224 wait complete. 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: [09/Mar/2024:00:06:51] ENGINE Bus STARTING 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: [09/Mar/2024:00:06:51] ENGINE Serving on http://172.21.15.73:8765 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: [09/Mar/2024:00:06:51] ENGINE Serving on https://172.21.15.73:7150 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: [09/Mar/2024:00:06:51] ENGINE Bus STARTED 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: mgrmap e22: x(active, since 1.02083s), standbys: y, z 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:53.130 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:52 smithi073 ceph-mon[32309]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:53.131 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > START_TIME' 2024-03-09T00:06:53.160 INFO:teuthology.run_tasks:Running task full_sequential_finally... 2024-03-09T00:06:53.170 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-09T00:06:53.209 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:06:53.209 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-09T00:06:53.212 INFO:teuthology.run_tasks:Running task workunit... 2024-03-09T00:06:53.263 INFO:tasks.workunit:Pulling workunits from ref 7d63482aa89628d733e298984a762389b625c761 2024-03-09T00:06:53.263 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-09T00:06:53.263 INFO:tasks.workunit:timeout=3h 2024-03-09T00:06:53.263 INFO:tasks.workunit:cleanup=True 2024-03-09T00:06:53.264 DEBUG:teuthology.orchestra.run.smithi073:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: [09/Mar/2024:00:06:51] ENGINE Bus STARTING 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: [09/Mar/2024:00:06:51] ENGINE Serving on http://172.21.15.73:8765 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: [09/Mar/2024:00:06:51] ENGINE Serving on https://172.21.15.73:7150 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: [09/Mar/2024:00:06:51] ENGINE Bus STARTED 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: mgrmap e22: x(active, since 1.02083s), standbys: y, z 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:53.265 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:52 smithi102 ceph-mon[36106]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:53.279 INFO:teuthology.orchestra.run.smithi073.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 77h/119d Inode: 1099511628283 Links: 2 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-03-09 00:04:41.372910555 +0000 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-03-09 00:04:41.372910555 +0000 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-03-09 00:06:36.420896722 +0000 2024-03-09T00:06:53.280 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: 2024-03-09 00:04:41.372910555 +0000 2024-03-09T00:06:53.280 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-09T00:06:53.281 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-09T00:06:53.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: [09/Mar/2024:00:06:51] ENGINE Bus STARTING 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: [09/Mar/2024:00:06:51] ENGINE Serving on http://172.21.15.73:8765 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: [09/Mar/2024:00:06:51] ENGINE Serving on https://172.21.15.73:7150 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: [09/Mar/2024:00:06:51] ENGINE Bus STARTED 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: mgrmap e22: x(active, since 1.02083s), standbys: y, z 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: from='client.15582 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule add", "fs": "cephfs", "path": "/", "snap_schedule": "1m", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:53.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:52 smithi179 ceph-mon[36637]: pgmap v3: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:53.354 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 7d63482aa89628d733e298984a762389b625c761 2024-03-09T00:06:53.401 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88444]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:53.401 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88444]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:53.401 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88444]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:53.401 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88444]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:53.409 INFO:tasks.workunit.client.0.smithi073.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-09T00:06:53.901 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88487]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:53.901 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88487]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:53.901 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88487]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:53.901 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:06:53 smithi073 sudo[88487]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:54.181 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:53 smithi102 sudo[68415]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:54.181 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:53 smithi102 sudo[68415]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:54.181 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:53 smithi102 sudo[68415]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:54.182 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:53 smithi102 sudo[68415]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.182 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.183 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.183 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:54 smithi102 ceph-mon[36106]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-09T00:06:54.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:54 smithi179 ceph-mon[36637]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='client.15624 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule retention add", "fs": "cephfs", "path": "/", "retention_spec_or_period": "6m3h", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='client.15630 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: pgmap v4: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:54.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:54 smithi073 ceph-mon[32309]: mgrmap e23: x(active, since 3s), standbys: y, z 2024-03-09T00:06:54.508 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:54 smithi102 sudo[68438]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:54.509 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:54 smithi102 sudo[68438]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:54.509 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:54 smithi102 sudo[68438]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:54.509 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:06:54 smithi102 sudo[68438]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:54.725 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69677]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:54.725 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69677]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:54.725 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69677]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:54.725 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69677]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:55.067 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69890]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:55.067 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69890]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:55.067 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69890]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:55.067 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:06:54 smithi179 sudo[69890]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:55.280 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89189]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:55.280 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89189]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:55.280 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89189]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:55.280 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89189]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:55.280 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89427]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:55.281 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89427]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:55.281 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89427]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.281 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.282 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:55.283 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:55 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.333 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:55.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:55.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:55 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='client.15636 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:55.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:55 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:55.651 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:06:55 smithi073 sudo[89427]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:55.785 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69305]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:55.785 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69305]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:55.785 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69305]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:55.785 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69305]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:56.070 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69308]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:56.070 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69308]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:56.070 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69308]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:56.070 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:06:55 smithi102 sudo[69308]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:56.293 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70421]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:56.293 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70421]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:56.293 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70421]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:56.293 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70421]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-09T00:06:56.294 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.295 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:56.296 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:56 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:06:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:06:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:06:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:06:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:56.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:56.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:56 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi073:/etc/ceph/ceph.conf 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi102:/etc/ceph/ceph.conf 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi179:/etc/ceph/ceph.conf 2024-03-09T00:06:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.conf 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi073:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi179:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi102:/etc/ceph/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: pgmap v5: 129 pgs: 129 active+clean; 1.1 MiB data, 389 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: Updating smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/config/ceph.client.admin.keyring 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: mgrmap e24: x(active, since 4s), standbys: y, z 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:06:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:56 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:06:56.581 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70456]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:56.581 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70456]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:56.581 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70456]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:56.581 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:06:56 smithi179 sudo[70456]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:56.797 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89582]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:56.797 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89582]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:56.797 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89582]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:56.797 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89582]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:57.151 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89585]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:57.151 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89585]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:57.151 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89585]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:57.151 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:06:56 smithi073 sudo[89585]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:57.299 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69343]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:57.299 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69343]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:57.299 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69343]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:57.300 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69343]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:57.758 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69346]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:57.758 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69346]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:57.759 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69346]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:57.759 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:06:57 smithi102 sudo[69346]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:57.801 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70461]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:57.801 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70461]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:57.801 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70461]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:57.801 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70461]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:58.072 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70464]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:58.073 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70464]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:58.073 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70464]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:58.073 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:06:57 smithi179 sudo[70464]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:58.326 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:58 smithi073 ceph-mon[32309]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-09T00:06:58.327 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89599]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:58.327 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89599]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:58.327 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89599]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:58.327 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89599]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:58.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:06:58 smithi179 ceph-mon[36637]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-09T00:06:58.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:06:58 smithi102 ceph-mon[36106]: pgmap v6: 129 pgs: 129 active+clean; 1.1 MiB data, 390 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 90 KiB/s wr, 66 op/s 2024-03-09T00:06:58.651 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89602]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:58.651 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89602]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:58.651 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89602]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:58.651 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:06:58 smithi073 sudo[89602]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:58.824 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69349]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:58.824 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69349]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:58.824 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69349]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:58.824 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69349]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:59.258 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69352]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:59.259 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69352]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:59.259 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69352]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:59.259 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:06:58 smithi102 sudo[69352]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:59.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70467]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:59.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70467]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:59.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70467]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:59.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70467]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:59.581 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70470]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:06:59.581 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70470]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:59.582 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70470]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:59.582 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:06:59 smithi179 sudo[70470]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:06:59.832 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89606]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:06:59.832 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89606]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:06:59.832 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89606]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:06:59.832 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89606]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:00.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89609]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:07:00.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89609]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:07:00.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89609]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:07:00.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:06:59 smithi073 sudo[89609]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70473]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:07:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70473]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:07:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70473]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:07:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70473]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:00.597 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 300 KiB/s wr, 88 op/s 2024-03-09T00:07:00.597 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69355]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69355]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69355]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:07:00.598 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69355]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 300 KiB/s wr, 88 op/s 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:07:00.425+0000 7fbed5c12700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:07:00.425+0000 7fbed5c12700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:07:00.425+0000 7fbed5c12700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:07:00.425+0000 7fbed5c12700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:07:00.652 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:07:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:07:00.425+0000 7fbed5c12700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-09T00:07:00.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70476]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:07:00.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70476]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70476]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 sudo[70476]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: pgmap v7: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 300 KiB/s wr, 88 op/s 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:00.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:01.008 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69358]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-09T00:07:01.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69358]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-09T00:07:01.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69358]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-09T00:07:01.009 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:00 smithi102 sudo[69358]: pam_unix(sudo:session): session closed for user root 2024-03-09T00:07:01.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.652 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:01 smithi073 ceph-mon[32309]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 233 KiB/s wr, 68 op/s 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.759 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:01 smithi102 ceph-mon[36106]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 233 KiB/s wr, 68 op/s 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-09T00:07:01.832 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-09T00:07:01.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-09T00:07:01.833 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:01 smithi179 ceph-mon[36637]: pgmap v8: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 76 KiB/s rd, 233 KiB/s wr, 68 op/s 2024-03-09T00:07:02.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:02 smithi073 ceph-mon[32309]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-03-09T00:07:02.758 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:02 smithi102 ceph-mon[36106]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-03-09T00:07:02.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:02 smithi179 ceph-mon[36637]: mgrmap e25: x(active, since 10s), standbys: y, z 2024-03-09T00:07:03.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:03 smithi073 ceph-mon[32309]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 239 KiB/s wr, 75 op/s 2024-03-09T00:07:03.758 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:03 smithi102 ceph-mon[36106]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 239 KiB/s wr, 75 op/s 2024-03-09T00:07:03.831 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:03 smithi179 ceph-mon[36637]: pgmap v9: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 74 KiB/s rd, 239 KiB/s wr, 75 op/s 2024-03-09T00:07:06.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:06 smithi179 ceph-mon[36637]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 219 KiB/s wr, 69 op/s 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.354+0000 7fec38a30640 1 -- 172.21.15.73:0/3527369260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34069310 msgr2=0x7fec3410e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.354+0000 7fec38a30640 1 --2- 172.21.15.73:0/3527369260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34069310 0x7fec3410e180 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7fec2c009ca0 tx=0x7fec2c02c180 comp rx=0 tx=0).stop 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 -- 172.21.15.73:0/3527369260 shutdown_connections 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 --2- 172.21.15.73:0/3527369260 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34069310 0x7fec3410e180 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 --2- 172.21.15.73:0/3527369260 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec34068970 0x7fec34068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 --2- 172.21.15.73:0/3527369260 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec341073e0 0x7fec341077c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.350 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 -- 172.21.15.73:0/3527369260 >> 172.21.15.73:0/3527369260 conn(0x7fec34077040 msgr2=0x7fec34077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 -- 172.21.15.73:0/3527369260 shutdown_connections 2024-03-09T00:07:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 -- 172.21.15.73:0/3527369260 wait complete. 2024-03-09T00:07:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.355+0000 7fec38a30640 1 Processor -- start 2024-03-09T00:07:06.351 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 -- start start 2024-03-09T00:07:06.352 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.352 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec34069310 0x7fec34100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.352 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec341073e0 0x7fec34100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.352 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec34111650 con 0x7fec34068970 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341114d0 con 0x7fec341073e0 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec38a30640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341117d0 con 0x7fec34069310 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38008/0 (socket says 172.21.15.73:38008) 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 -- 172.21.15.73:0/3585275497 learned_addr learned my addr 172.21.15.73:0/3585275497 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:06.356 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec29d9b640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec34069310 0x7fec34100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec32d9d640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec341073e0 0x7fec34100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 -- 172.21.15.73:0/3585275497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec34069310 msgr2=0x7fec34100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec34069310 0x7fec34100a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 -- 172.21.15.73:0/3585275497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec341073e0 msgr2=0x7fec34100f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec341073e0 0x7fec34100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 -- 172.21.15.73:0/3585275497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec341a5f50 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.356+0000 7fec3259c640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fec2400c890 tx=0x7fec2400cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.357+0000 7fec2bfff640 1 -- 172.21.15.73:0/3585275497 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24004220 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.357+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec341a6240 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.357+0000 7fec2bfff640 1 -- 172.21.15.73:0/3585275497 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec240043c0 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.357+0000 7fec2bfff640 1 -- 172.21.15.73:0/3585275497 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24004a00 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.357+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec341a67e0 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.359+0000 7fec2bfff640 1 -- 172.21.15.73:0/3585275497 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fec240079b0 con 0x7fec34068970 2024-03-09T00:07:06.357 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.359+0000 7fec3b60b740 1 Processor -- start 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.359+0000 7fec3b60b740 1 -- start start 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee0a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28eec940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28e455f0 con 0x559e28ee0fd0 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28dee930 con 0x559e28ee1d40 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3b60b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28e41690 con 0x559e28eea550 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec2bfff640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 0x7fec200a5020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38020/0 (socket says 172.21.15.73:38020) 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 -- 172.21.15.73:0/2137961673 learned_addr learned my addr 172.21.15.73:0/2137961673 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec32d9d640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 0x7fec200a5020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.358 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec32d9d640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee0a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec29d9b640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28eec940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec2bfff640 1 -- 172.21.15.73:0/3585275497 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fec240cf550 con 0x7fec34068970 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 -- 172.21.15.73:0/2137961673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 msgr2=0x559e28eec940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28eec940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 -- 172.21.15.73:0/2137961673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 msgr2=0x559e28ee0a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee0a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 -- 172.21.15.73:0/2137961673 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e28b5a210 con 0x559e28ee0fd0 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec29d9b640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28eec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec3259c640 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fec240a3840 tx=0x7fec24086100 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32ab629658fe3a47 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.360+0000 7fec32d9d640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 0x7fec200a5020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec2c004af0 tx=0x7fec2c032450 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.364 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec28d99640 1 -- 172.21.15.73:0/2137961673 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2401d070 con 0x559e28ee0fd0 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec28d99640 1 -- 172.21.15.73:0/2137961673 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec240a3d70 con 0x559e28ee0fd0 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec28d99640 1 -- 172.21.15.73:0/2137961673 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24007ca0 con 0x559e28ee0fd0 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 -- 172.21.15.73:0/2137961673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 msgr2=0x559e28ee1450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 crypto rx=0x7fec240a3840 tx=0x7fec24086100 comp rx=0 tx=0).stop 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 -- 172.21.15.73:0/2137961673 shutdown_connections 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28eec940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee1450 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2137961673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee0a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 -- 172.21.15.73:0/2137961673 >> 172.21.15.73:0/2137961673 conn(0x559e28ea85b0 msgr2=0x559e28edbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 -- 172.21.15.73:0/2137961673 shutdown_connections 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.361+0000 7fec3b60b740 1 -- 172.21.15.73:0/2137961673 wait complete. 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 -- start start 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 Processor -- start 2024-03-09T00:07:06.365 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee0fd0 0x559e28ee75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee7b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28eed220 con 0x559e28eea550 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28e01d90 con 0x559e28ee1d40 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559e28eed3a0 con 0x559e28ee0fd0 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3b60b740 10 client.0 _ll_register_callbacks cb 0x559e28b59a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec32d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec32d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38036/0 (socket says 172.21.15.73:38036) 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec32d9d640 1 -- 172.21.15.73:0/2650349892 learned_addr learned my addr 172.21.15.73:0/2650349892 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec29d9b640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee7b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.362+0000 7fec3259c640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee0fd0 0x559e28ee75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.366 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 -- 172.21.15.73:0/2650349892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee0fd0 msgr2=0x559e28ee75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee0fd0 0x559e28ee75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 -- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 msgr2=0x559e28ee7b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee7b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 -- 172.21.15.73:0/2650349892 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e28c26620 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec32d9d640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fec2c036000 tx=0x7fec2c045070 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea82ec86c8e4f510 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2c051020 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fec2c032bf0 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec240a3d70 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec2c05c050 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec3b60b740 10 client.15663 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559e28ef9560 con 0x559e28eea550 2024-03-09T00:07:06.367 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec240a3d70 con 0x559e28eea550 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.363+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec2c04c030 con 0x559e28eea550 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 10 client.15663 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 10 client.15663 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 20 client.15663 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28efaa70 0x559e28efce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 4 client.15663 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559e28e52b30 con 0x559e28efaa70 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3259c640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28efaa70 0x559e28efce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec19ffb640 10 client.15663 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.364+0000 7fec3259c640 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28efaa70 0x559e28efce60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca1a1bca47c7c62f server_cookie=95f9e0b0b837eb8c in_seq=0 out_seq=0 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec19ffb640 1 -- 172.21.15.73:0/2650349892 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559e28e52b30 con 0x559e28efaa70 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec19ffb640 10 client.15663 handle_command_reply: tid=0 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 client.15663 shutdown 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 msgr2=0x559e28ee8070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.368 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 crypto rx=0x7fec2c036000 tx=0x7fec2c045070 comp rx=0 tx=0).stop 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 shutdown_connections 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28efaa70 0x559e28efce60 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28eea550 0x559e28ee8070 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ee7b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec19ffb640 0 client.15663 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 --2- 172.21.15.73:0/2650349892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee0fd0 0x559e28ee75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 >> 172.21.15.73:0/2650349892 conn(0x559e28ea85b0 msgr2=0x559e28edda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 shutdown_connections 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 1 -- 172.21.15.73:0/2650349892 wait complete. 2024-03-09T00:07:06.369 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.365+0000 7fec3b60b740 20 client.15663 trim_cache size 0 max 16384 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.378+0000 7fec3b60b740 1 Processor -- start 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.378+0000 7fec3b60b740 1 -- start start 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ef1610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28ef1b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.377 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341ae900 con 0x559e28ee0fd0 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341b0ec0 con 0x559e28ee1d40 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3b60b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341aef00 con 0x559e28eea550 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38040/0 (socket says 172.21.15.73:38040) 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 -- 172.21.15.73:0/3743652854 learned_addr learned my addr 172.21.15.73:0/3743652854 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec32d9d640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28ef1b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec29d9b640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ef1610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 -- 172.21.15.73:0/3743652854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 msgr2=0x559e28ef1b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28ef1b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 -- 172.21.15.73:0/3743652854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 msgr2=0x559e28ef1610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ef1610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 -- 172.21.15.73:0/3743652854 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559e28ee02a0 con 0x559e28ee0fd0 2024-03-09T00:07:06.378 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec32d9d640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28ef1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec29d9b640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ef1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec3259c640 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fec24097a30 tx=0x7fec240a5040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4400b87c06d67ee server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec1bfff640 1 -- 172.21.15.73:0/3743652854 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec240b3070 con 0x559e28ee0fd0 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec1bfff640 1 -- 172.21.15.73:0/3743652854 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec2409f720 con 0x559e28ee0fd0 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.379+0000 7fec1bfff640 1 -- 172.21.15.73:0/3743652854 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec2401d040 con 0x559e28ee0fd0 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- 172.21.15.73:0/3743652854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 msgr2=0x559e28ee9d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 crypto rx=0x7fec24097a30 tx=0x7fec240a5040 comp rx=0 tx=0).stop 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- 172.21.15.73:0/3743652854 shutdown_connections 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28eea550 0x559e28ef1b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28ee1d40 0x559e28ef1610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- 172.21.15.73:0/3743652854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ee9d20 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- 172.21.15.73:0/3743652854 >> 172.21.15.73:0/3743652854 conn(0x559e28ea85b0 msgr2=0x559e28edb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- 172.21.15.73:0/3743652854 shutdown_connections 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- 172.21.15.73:0/3743652854 wait complete. 2024-03-09T00:07:06.379 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- start start 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 Processor -- start 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee1d40 0x559e28eede00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28eea550 0x559e28eee410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341af540 con 0x559e28ee0fd0 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341b0860 con 0x559e28eea550 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.380+0000 7fec3b60b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec341afba0 con 0x559e28ee1d40 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3b60b740 10 client.0 _ll_register_callbacks cb 0x7fec2409b110 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38052/0 (socket says 172.21.15.73:38052) 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 -- 172.21.15.73:0/4054842354 learned_addr learned my addr 172.21.15.73:0/4054842354 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec32d9d640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28eea550 0x559e28eee410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 -- 172.21.15.73:0/4054842354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee1d40 msgr2=0x559e28eede00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.380 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee1d40 0x559e28eede00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 -- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28eea550 msgr2=0x559e28eee410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28eea550 0x559e28eee410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 -- 172.21.15.73:0/4054842354 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559e28eeec10 con 0x559e28ee0fd0 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3259c640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fec2409b560 tx=0x7fec2409b4d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4d37161f826b300 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec19ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:06.381 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fec24065070 con 0x559e28ee0fd0 2024-03-09T00:07:06.382 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fec24097c40 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fec2409f720 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec2401d040 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3b60b740 10 client.15669 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559e28f52070 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.381+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fec2409f720 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fec240b42f0 con 0x559e28ee0fd0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 10 client.15669 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 10 client.15669 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 20 client.15669 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28f522f0 0x559e28f546e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 4 client.15669 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559e28e4ce80 con 0x559e28f522f0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec29d9b640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28f522f0 0x559e28f546e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:06.383 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec29d9b640 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28f522f0 0x559e28f546e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c65f167b0000cc86 server_cookie=8cf8a3307b6359aa in_seq=0 out_seq=0 2024-03-09T00:07:06.384 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.382+0000 7fec19ffb640 10 client.15669 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:06.395 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec19ffb640 1 -- 172.21.15.73:0/4054842354 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559e28e4ce80 con 0x559e28f522f0 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec19ffb640 10 client.15669 handle_command_reply: tid=0 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 client.15669 shutdown 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 msgr2=0x559e28ef1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 crypto rx=0x7fec2409b560 tx=0x7fec2409b4d0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 shutdown_connections 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559e28f522f0 0x559e28f546e0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559e28eea550 0x559e28eee410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559e28ee1d40 0x559e28eede00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec19ffb640 0 client.15669 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 --2- 172.21.15.73:0/4054842354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559e28ee0fd0 0x559e28ef1a80 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.396 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 >> 172.21.15.73:0/4054842354 conn(0x559e28ea85b0 msgr2=0x559e28ee5ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 shutdown_connections 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 1 -- 172.21.15.73:0/4054842354 wait complete. 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.396+0000 7fec3b60b740 20 client.15669 trim_cache size 0 max 16384 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:07:06.397 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 msgr2=0x7fec200a5020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 0x7fec200a5020 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fec2c004af0 tx=0x7fec2c032450 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 msgr2=0x7fec34100510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 crypto rx=0x7fec2400c890 tx=0x7fec2400cd50 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 shutdown_connections 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fec341073e0 0x7fec34100f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fec34069310 0x7fec34100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fec200a2b60 0x7fec200a5020 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 --2- 172.21.15.73:0/3585275497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fec34068970 0x7fec34100510 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:06.399 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 >> 172.21.15.73:0/3585275497 conn(0x7fec34077040 msgr2=0x7fec340ff030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:06.400 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 shutdown_connections 2024-03-09T00:07:06.400 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:06.397+0000 7fec38a30640 1 -- 172.21.15.73:0/3585275497 wait complete. 2024-03-09T00:07:06.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:06 smithi073 ceph-mon[32309]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 219 KiB/s wr, 69 op/s 2024-03-09T00:07:06.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:06 smithi102 ceph-mon[36106]: pgmap v10: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 67 KiB/s rd, 219 KiB/s wr, 69 op/s 2024-03-09T00:07:08.151 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:08 smithi073 ceph-mon[32309]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 221 KiB/s wr, 72 op/s 2024-03-09T00:07:08.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:08 smithi179 ceph-mon[36637]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 221 KiB/s wr, 72 op/s 2024-03-09T00:07:08.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:08 smithi102 ceph-mon[36106]: pgmap v11: 129 pgs: 129 active+clean; 1.3 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 69 KiB/s rd, 221 KiB/s wr, 72 op/s 2024-03-09T00:07:10.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:10 smithi179 ceph-mon[36637]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 184 KiB/s wr, 47 op/s 2024-03-09T00:07:10.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:10 smithi073 ceph-mon[32309]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 184 KiB/s wr, 47 op/s 2024-03-09T00:07:10.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:10 smithi102 ceph-mon[36106]: pgmap v12: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 51 KiB/s rd, 184 KiB/s wr, 47 op/s 2024-03-09T00:07:12.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:12 smithi179 ceph-mon[36637]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 23 op/s 2024-03-09T00:07:12.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:12 smithi073 ceph-mon[32309]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 23 op/s 2024-03-09T00:07:12.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:12 smithi102 ceph-mon[36106]: pgmap v13: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 23 op/s 2024-03-09T00:07:13.712 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:07:13.712 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:07:13.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 -- 172.21.15.73:0/3776552865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a0100460 msgr2=0x7f65a0100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 --2- 172.21.15.73:0/3776552865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a0100460 0x7f65a0100840 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 crypto rx=0x7f6590009900 tx=0x7f659002bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:13.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 -- 172.21.15.73:0/3776552865 shutdown_connections 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 --2- 172.21.15.73:0/3776552865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01054e0 0x7f65a010d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 --2- 172.21.15.73:0/3776552865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100e10 0x7f65a0104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 --2- 172.21.15.73:0/3776552865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a0100460 0x7f65a0100840 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 -- 172.21.15.73:0/3776552865 >> 172.21.15.73:0/3776552865 conn(0x7f65a0077350 msgr2=0x7f65a0077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 -- 172.21.15.73:0/3776552865 shutdown_connections 2024-03-09T00:07:13.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.835+0000 7f65a83da640 1 -- 172.21.15.73:0/3776552865 wait complete. 2024-03-09T00:07:13.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 Processor -- start 2024-03-09T00:07:13.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 -- start start 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 0x7f65a0198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 0x7f65a01a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0110590 con 0x7f65a01054e0 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0110410 con 0x7f65a01994e0 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a83da640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a0110710 con 0x7f65a0100460 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38912/0 (socket says 172.21.15.73:38912) 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 -- 172.21.15.73:0/491260824 learned_addr learned my addr 172.21.15.73:0/491260824 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a6176640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 0x7f65a0198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a6977640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 0x7f65a01a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 -- 172.21.15.73:0/491260824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 msgr2=0x7f65a0198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 0x7f65a0198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 -- 172.21.15.73:0/491260824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 msgr2=0x7f65a01a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 0x7f65a01a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.836+0000 7f65a5975640 1 -- 172.21.15.73:0/491260824 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65a01a4260 con 0x7f65a01054e0 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65a6977640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 0x7f65a01a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65a6176640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 0x7f65a0198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65a5975640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f659c00d740 tx=0x7f659c00dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65977fe640 1 -- 172.21.15.73:0/491260824 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c017070 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65a01a4550 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65977fe640 1 -- 172.21.15.73:0/491260824 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f659c00de50 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65977fe640 1 -- 172.21.15.73:0/491260824 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c004e00 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.837+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65a01a4aa0 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.839+0000 7f65977fe640 1 -- 172.21.15.73:0/491260824 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f659c004fa0 con 0x7f65a01054e0 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.839+0000 7f65aafb0740 1 Processor -- start 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 -- start start 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65977fe640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 0x7f658c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0beb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.846 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0ca050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ab021e00 con 0x5573ab0bf080 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573aafcab10 con 0x5573ab0c7c60 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65aafb0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ab01dea0 con 0x5573ab0be760 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65977fe640 1 -- 172.21.15.73:0/491260824 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f659c0cef10 con 0x7f65a01054e0 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a6977640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 0x7f658c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.847 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38918/0 (socket says 172.21.15.73:38918) 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 -- 172.21.15.73:0/1533913563 learned_addr learned my addr 172.21.15.73:0/1533913563 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a6176640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0beb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a6977640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0ca050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 -- 172.21.15.73:0/1533913563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 msgr2=0x5573ab0beb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0beb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 -- 172.21.15.73:0/1533913563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 msgr2=0x5573ab0ca050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0ca050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a5975640 1 -- 172.21.15.73:0/1533913563 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573aae007d0 con 0x5573ab0bf080 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.840+0000 7f65a6977640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:13.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.841+0000 7f65a5975640 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f659c0cc8d0 tx=0x7f659c09ae90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=68360330c34070a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.841+0000 7f65a6977640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 0x7f658c0a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x5573ab0ba020 tx=0x7f6598009830 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.841+0000 7f6594bfe640 1 -- 172.21.15.73:0/1533913563 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c020070 con 0x5573ab0bf080 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.841+0000 7f6594bfe640 1 -- 172.21.15.73:0/1533913563 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f659c09b690 con 0x5573ab0bf080 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.841+0000 7f6594bfe640 1 -- 172.21.15.73:0/1533913563 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c00ee50 con 0x5573ab0bf080 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- 172.21.15.73:0/1533913563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 msgr2=0x5573ab0c7720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 crypto rx=0x7f659c0cc8d0 tx=0x7f659c09ae90 comp rx=0 tx=0).stop 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- 172.21.15.73:0/1533913563 shutdown_connections 2024-03-09T00:07:13.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0ca050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c7720 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- 172.21.15.73:0/1533913563 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0beb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- 172.21.15.73:0/1533913563 >> 172.21.15.73:0/1533913563 conn(0x5573ab0b8e90 msgr2=0x5573ab0b9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- 172.21.15.73:0/1533913563 shutdown_connections 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- 172.21.15.73:0/1533913563 wait complete. 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- start start 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 Processor -- start 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0c4d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ab0ca990 con 0x5573ab0bf080 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573aafde0a0 con 0x5573ab0c7c60 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5573ab0cab10 con 0x5573ab0be760 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65aafb0740 10 client.0 _ll_register_callbacks cb 0x5573aafdda70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65a5975640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38934/0 (socket says 172.21.15.73:38934) 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.842+0000 7f65a5975640 1 -- 172.21.15.73:0/931778512 learned_addr learned my addr 172.21.15.73:0/931778512 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a6176640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 -- 172.21.15.73:0/931778512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 msgr2=0x5573ab0c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0c7440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 -- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 msgr2=0x5573ab0c4d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0c4d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 -- 172.21.15.73:0/931778512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573aad363f0 con 0x5573ab0bf080 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65a5975640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f659c09ac90 tx=0x7f659c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6ef532b38c0ca72 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:13.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f659c020050 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f659c0cc640 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f659c09b690 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f659c09f0e0 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65aafb0740 10 client.15681 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5573aacf60f0 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f659c093070 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.843+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f659c096ce0 con 0x5573ab0bf080 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 10 client.15681 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 10 client.15681 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:13.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 20 client.15681 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab0d9960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 4 client.15681 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5573ab02f340 con 0x5573ab0d7570 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65a6176640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab0d9960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f65a6176640 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab0d9960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc8d8fea71946e7a server_cookie=211a0b5165525daa in_seq=0 out_seq=0 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.844+0000 7f6589ffb640 10 client.15681 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f6589ffb640 1 -- 172.21.15.73:0/931778512 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5573ab02f340 con 0x5573ab0d7570 2024-03-09T00:07:13.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f6589ffb640 10 client.15681 handle_command_reply: tid=0 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 client.15681 shutdown 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 msgr2=0x5573ab0c47d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 crypto rx=0x7f659c09ac90 tx=0x7f659c00ee80 comp rx=0 tx=0).stop 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 shutdown_connections 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab0d9960 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0c7c60 0x5573ab0c4d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f6589ffb640 0 client.15681 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:13.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0bf080 0x5573ab0c47d0 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 --2- 172.21.15.73:0/931778512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 >> 172.21.15.73:0/931778512 conn(0x5573ab0b8e90 msgr2=0x5573ab0babe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 shutdown_connections 2024-03-09T00:07:13.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 1 -- 172.21.15.73:0/931778512 wait complete. 2024-03-09T00:07:13.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.845+0000 7f65aafb0740 20 client.15681 trim_cache size 0 max 16384 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 Processor -- start 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 -- start start 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0c7440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0c7c60 0x5573ab0ced50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01ac6c0 con 0x5573ab0be760 2024-03-09T00:07:13.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01aec80 con 0x5573ab0bf080 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65aafb0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01accc0 con 0x5573ab0c7c60 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65a6176640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65a6176640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38936/0 (socket says 172.21.15.73:38936) 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.860+0000 7f65a6176640 1 -- 172.21.15.73:0/3140716504 learned_addr learned my addr 172.21.15.73:0/3140716504 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 -- 172.21.15.73:0/3140716504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0c7c60 msgr2=0x5573ab0ced50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0c7c60 0x5573ab0ced50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a5975640 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0c7440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 -- 172.21.15.73:0/3140716504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 msgr2=0x5573ab0c7440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0c7440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 -- 172.21.15.73:0/3140716504 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5573ab0d9dc0 con 0x5573ab0be760 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a5975640 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:13.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65a6176640 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6590009900 tx=0x7f659002c900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32175149e9571412 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f658bfff640 1 -- 172.21.15.73:0/3140716504 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590045030 con 0x5573ab0be760 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f658bfff640 1 -- 172.21.15.73:0/3140716504 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6590046b80 con 0x5573ab0be760 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f658bfff640 1 -- 172.21.15.73:0/3140716504 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6590005330 con 0x5573ab0be760 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 -- 172.21.15.73:0/3140716504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 msgr2=0x5573ab0c6f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 crypto rx=0x7f6590009900 tx=0x7f659002c900 comp rx=0 tx=0).stop 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 -- 172.21.15.73:0/3140716504 shutdown_connections 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0c7c60 0x5573ab0ced50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0c7440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 --2- 172.21.15.73:0/3140716504 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0be760 0x5573ab0c6f00 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.861+0000 7f65aafb0740 1 -- 172.21.15.73:0/3140716504 >> 172.21.15.73:0/3140716504 conn(0x5573ab0b8e90 msgr2=0x5573ab0c4790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- 172.21.15.73:0/3140716504 shutdown_connections 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- 172.21.15.73:0/3140716504 wait complete. 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- start start 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 Processor -- start 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0ced00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0cf240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01ad300 con 0x5573ab0c7c60 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01ae620 con 0x5573ab0bf080 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65a01ad960 con 0x5573ab0be760 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65aafb0740 10 client.0 _ll_register_callbacks cb 0x7f6590009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:13.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65a6977640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65a6977640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38952/0 (socket says 172.21.15.73:38952) 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65a6977640 1 -- 172.21.15.73:0/895431371 learned_addr learned my addr 172.21.15.73:0/895431371 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65a5975640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0cf240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.862+0000 7f65a6176640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0ced00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 -- 172.21.15.73:0/895431371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 msgr2=0x5573ab0ced00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0ced00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 -- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 msgr2=0x5573ab0cf240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0cf240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 -- 172.21.15.73:0/895431371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5573ab0cdaa0 con 0x5573ab0c7c60 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65a6977640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6598010700 tx=0x7f65980097c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc7eb40fd2c70b2a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6598023020 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f659801f830 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6590046b80 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f659802e050 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 10 client.15687 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5573ab12e360 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6590046b80 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6598015b80 con 0x5573ab0c7c60 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 10 client.15687 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 10 client.15687 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 20 client.15687 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:13.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab130b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 4 client.15687 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.863+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5573ab029690 con 0x5573ab0d7570 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65a6176640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab130b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f6589ffb640 10 client.15687 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65a6176640 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab130b50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e42b290f0a804784 server_cookie=ff4fb8e718adc171 in_seq=0 out_seq=0 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f6589ffb640 1 -- 172.21.15.73:0/895431371 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5573ab029690 con 0x5573ab0d7570 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f6589ffb640 10 client.15687 handle_command_reply: tid=0 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 client.15687 shutdown 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 msgr2=0x5573ab0cd2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 crypto rx=0x7f6598010700 tx=0x7f65980097c0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 shutdown_connections 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5573ab0d7570 0x5573ab130b50 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5573ab0c7c60 0x5573ab0cd2a0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f6589ffb640 0 client.15687 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5573ab0bf080 0x5573ab0cf240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 --2- 172.21.15.73:0/895431371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5573ab0be760 0x5573ab0ced00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 >> 172.21.15.73:0/895431371 conn(0x5573ab0b8e90 msgr2=0x5573ab0c2f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.864+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 shutdown_connections 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65aafb0740 1 -- 172.21.15.73:0/895431371 wait complete. 2024-03-09T00:07:13.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65aafb0740 20 client.15687 trim_cache size 0 max 16384 2024-03-09T00:07:13.872 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:13.872 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:07:13.872 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:07:13.872 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 msgr2=0x7f658c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 0x7f658c0a4f50 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x5573ab0ba020 tx=0x7f6598009830 comp rx=0 tx=0).stop 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 msgr2=0x7f65a0198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 crypto rx=0x7f659c00d740 tx=0x7f659c00dc00 comp rx=0 tx=0).stop 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 shutdown_connections 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f65a01994e0 0x7f65a01a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f658c0a2a90 0x7f658c0a4f50 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f65a01054e0 0x7f65a0198f50 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 --2- 172.21.15.73:0/491260824 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f65a0100460 0x7f65a0198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 >> 172.21.15.73:0/491260824 conn(0x7f65a0077350 msgr2=0x7f65a00fe460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 shutdown_connections 2024-03-09T00:07:13.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.865+0000 7f65a83da640 1 -- 172.21.15.73:0/491260824 wait complete. 2024-03-09T00:07:13.875 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:07:13.876 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:07:13.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 -- 172.21.15.73:0/1241090329 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100e10 msgr2=0x7f3e30104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 --2- 172.21.15.73:0/1241090329 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100e10 0x7f3e30104e50 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 crypto rx=0x7f3e20009900 tx=0x7f3e2002bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 -- 172.21.15.73:0/1241090329 shutdown_connections 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 --2- 172.21.15.73:0/1241090329 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 0x7f3e3010d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 --2- 172.21.15.73:0/1241090329 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100e10 0x7f3e30104e50 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 --2- 172.21.15.73:0/1241090329 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e30100460 0x7f3e30100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 -- 172.21.15.73:0/1241090329 >> 172.21.15.73:0/1241090329 conn(0x7f3e300772f0 msgr2=0x7f3e30077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 -- 172.21.15.73:0/1241090329 shutdown_connections 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.984+0000 7f3e368a0640 1 -- 172.21.15.73:0/1241090329 wait complete. 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 Processor -- start 2024-03-09T00:07:13.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 -- start start 2024-03-09T00:07:13.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 0x7f3e3019cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e301968f0 0x7f3e30196da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30110610 con 0x7f3e30100460 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30110490 con 0x7f3e301968f0 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e368a0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e30110790 con 0x7f3e30105530 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e2ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38976/0 (socket says 172.21.15.73:38976) 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e2ffff640 1 -- 172.21.15.73:0/298839923 learned_addr learned my addr 172.21.15.73:0/298839923 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e2f7fe640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 0x7f3e3019cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.985+0000 7f3e34e3d640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e301968f0 0x7f3e30196da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 -- 172.21.15.73:0/298839923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 msgr2=0x7f3e3019cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 0x7f3e3019cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 -- 172.21.15.73:0/298839923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e301968f0 msgr2=0x7f3e30196da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:13.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e301968f0 0x7f3e30196da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 -- 172.21.15.73:0/298839923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e30197410 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2ffff640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c920 tx=0x7f3e2400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24007c70 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e30197700 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e24007e10 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e24018890 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.986+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e301a3900 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.987+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e300680e0 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.988+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f3e24018a70 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.988+0000 7f3e2d7fa640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 0x7f3e180a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.989+0000 7f3e2f7fe640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 0x7f3e180a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.989+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3e240cf690 con 0x7f3e30100460 2024-03-09T00:07:13.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.989+0000 7f3e2f7fe640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 0x7f3e180a4f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e2002c240 tx=0x7f3e2002df90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:13.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:13.993+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3e24014030 con 0x7f3e30100460 2024-03-09T00:07:14.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.210+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3e301087c0 con 0x7f3e30100460 2024-03-09T00:07:14.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.211+0000 7f3e2d7fa640 1 -- 172.21.15.73:0/298839923 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f3e2400de20 con 0x7f3e30100460 2024-03-09T00:07:14.206 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:07:14.206 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:07:14.206 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 msgr2=0x7f3e180a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 0x7f3e180a4f50 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f3e2002c240 tx=0x7f3e2002df90 comp rx=0 tx=0).stop 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 msgr2=0x7f3e3019c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 crypto rx=0x7f3e2400c920 tx=0x7f3e2400cde0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 shutdown_connections 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3e301968f0 0x7f3e30196da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3e30105530 0x7f3e3019cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3e180a2a90 0x7f3e180a4f50 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 --2- 172.21.15.73:0/298839923 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3e30100460 0x7f3e3019c6f0 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 >> 172.21.15.73:0/298839923 conn(0x7f3e300772f0 msgr2=0x7f3e300fe240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 shutdown_connections 2024-03-09T00:07:14.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.213+0000 7f3e368a0640 1 -- 172.21.15.73:0/298839923 wait complete. 2024-03-09T00:07:14.219 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:07:14.219 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:07:14.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- 172.21.15.73:0/202610422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141003e0 msgr2=0x7f84141007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- 172.21.15.73:0/202610422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141003e0 0x7f84141007c0 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 crypto rx=0x7f8410009900 tx=0x7f841002bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- 172.21.15.73:0/202610422 shutdown_connections 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- 172.21.15.73:0/202610422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841410d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- 172.21.15.73:0/202610422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8414100d90 0x7f8414104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- 172.21.15.73:0/202610422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141003e0 0x7f84141007c0 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- 172.21.15.73:0/202610422 >> 172.21.15.73:0/202610422 conn(0x7f84140773a0 msgr2=0x7f84140777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- 172.21.15.73:0/202610422 shutdown_connections 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- 172.21.15.73:0/202610422 wait complete. 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 Processor -- start 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- start start 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 0x7f841419c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841419cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8414110560 con 0x7f84141968a0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141103e0 con 0x7f84141054b0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.329+0000 7f841c6f8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141106e0 con 0x7f84141003e0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38988/0 (socket says 172.21.15.73:38988) 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 -- 172.21.15.73:0/1876642395 learned_addr learned my addr 172.21.15.73:0/1876642395 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f8419c93640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841419cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841a494640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 0x7f841419c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 -- 172.21.15.73:0/1876642395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 msgr2=0x7f841419c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 0x7f841419c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 -- 172.21.15.73:0/1876642395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 msgr2=0x7f841419cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841419cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 -- 172.21.15.73:0/1876642395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84141973c0 con 0x7f84141968a0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f8419c93640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841a494640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 0x7f841419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841ac95640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f840c00cad0 tx=0x7f840c00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f840b7fe640 1 -- 172.21.15.73:0/1876642395 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c00ae20 con 0x7f84141968a0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84141976b0 con 0x7f84141968a0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f840b7fe640 1 -- 172.21.15.73:0/1876642395 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f840c005020 con 0x7f84141968a0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f840b7fe640 1 -- 172.21.15.73:0/1876642395 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840c00f6f0 con 0x7f84141968a0 2024-03-09T00:07:14.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.330+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84141a3900 con 0x7f84141968a0 2024-03-09T00:07:14.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:14 smithi179 ceph-mon[36637]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-09T00:07:14.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.332+0000 7f840b7fe640 1 -- 172.21.15.73:0/1876642395 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f840c0051d0 con 0x7f84141968a0 2024-03-09T00:07:14.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.332+0000 7f841f3bc740 1 Processor -- start 2024-03-09T00:07:14.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 -- start start 2024-03-09T00:07:14.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fe03370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fe038b0 0x55ce8fe05ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fdbf360 con 0x55ce8fdf7e60 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fd5bec0 con 0x55ce8fdf8780 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841f3bc740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fd04d30 con 0x55ce8fe038b0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f840b7fe640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 0x7f84000a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39002/0 (socket says 172.21.15.73:39002) 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 -- 172.21.15.73:0/3239981819 learned_addr learned my addr 172.21.15.73:0/3239981819 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841ac95640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 0x7f84000a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f841a494640 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fe03370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f840b7fe640 1 -- 172.21.15.73:0/1876642395 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f840c0cf570 con 0x7f84141968a0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fe038b0 0x55ce8fe05ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 -- 172.21.15.73:0/3239981819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fe038b0 msgr2=0x55ce8fe05ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fe038b0 0x55ce8fe05ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 -- 172.21.15.73:0/3239981819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 msgr2=0x55ce8fe03370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fe03370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.333+0000 7f8419c93640 1 -- 172.21.15.73:0/3239981819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce8fd03bc0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841ac95640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 0x7f84000a4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f840c0cc810 tx=0x7f840c01a040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f8419c93640 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8404004650 tx=0x7f84040096c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d1682426d2c1b66 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f8408ff9640 1 -- 172.21.15.73:0/3239981819 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840401a020 con 0x55ce8fdf7e60 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f8408ff9640 1 -- 172.21.15.73:0/3239981819 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404011b50 con 0x55ce8fdf7e60 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f8408ff9640 1 -- 172.21.15.73:0/3239981819 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404010ba0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 -- 172.21.15.73:0/3239981819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 msgr2=0x55ce8fdf8240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 crypto rx=0x7f8404004650 tx=0x7f84040096c0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 -- 172.21.15.73:0/3239981819 shutdown_connections 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fe038b0 0x55ce8fe05ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fe03370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 --2- 172.21.15.73:0/3239981819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fdf8240 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.334+0000 7f841f3bc740 1 -- 172.21.15.73:0/3239981819 >> 172.21.15.73:0/3239981819 conn(0x55ce8fdf28d0 msgr2=0x55ce8fdf2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- 172.21.15.73:0/3239981819 shutdown_connections 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- 172.21.15.73:0/3239981819 wait complete. 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- start start 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 Processor -- start 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe00d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe01270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fcfa4a0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fd57f60 con 0x55ce8fe038b0 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ce8fe066a0 con 0x55ce8fdf8780 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841f3bc740 10 client.0 _ll_register_callbacks cb 0x55ce8fd55d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39006/0 (socket says 172.21.15.73:39006) 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 -- 172.21.15.73:0/2066539790 learned_addr learned my addr 172.21.15.73:0/2066539790 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe01270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f841a494640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe00d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 -- 172.21.15.73:0/2066539790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 msgr2=0x55ce8fe00d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe00d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 -- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 msgr2=0x55ce8fe01270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.335+0000 7f8419c93640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe01270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f8419c93640 1 -- 172.21.15.73:0/2066539790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce8fa703f0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841a494640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe00d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f8419c93640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8404017020 tx=0x7f8404005d30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6b7a7e4740088c03 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840401a040 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8404010920 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8404011b50 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8404010b20 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 10 client.15705 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ce8fb3a7d0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404011b50 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f840400f040 con 0x55ce8fdf7e60 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 10 client.15705 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 10 client.15705 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 20 client.15705 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:14.340 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe138a0 0x55ce8fe15c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 4 client.15705 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.336+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ce8fd69400 con 0x55ce8fe138a0 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.337+0000 7f841a494640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe138a0 0x55ce8fe15c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.337+0000 7f83fdffb640 10 client.15705 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.337+0000 7f841a494640 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe138a0 0x55ce8fe15c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=22919e34820d3f6f server_cookie=cd28244fd832d20e in_seq=0 out_seq=0 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f83fdffb640 1 -- 172.21.15.73:0/2066539790 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ce8fd69400 con 0x55ce8fe138a0 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f83fdffb640 10 client.15705 handle_command_reply: tid=0 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 client.15705 shutdown 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 msgr2=0x55ce8fe007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 crypto rx=0x7f8404017020 tx=0x7f8404005d30 comp rx=0 tx=0).stop 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 shutdown_connections 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe138a0 0x55ce8fe15c90 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe01270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f83fdffb640 0 client.15705 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe00d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2066539790 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe007f0 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 >> 172.21.15.73:0/2066539790 conn(0x55ce8fdf28d0 msgr2=0x55ce8fdf4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 shutdown_connections 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 1 -- 172.21.15.73:0/2066539790 wait complete. 2024-03-09T00:07:14.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.338+0000 7f841f3bc740 20 client.15705 trim_cache size 0 max 16384 2024-03-09T00:07:14.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 Processor -- start 2024-03-09T00:07:14.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 -- start start 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe02f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fdfffa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141aa5d0 con 0x55ce8fe038b0 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141acb90 con 0x55ce8fdf8780 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f841f3bc740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141aabd0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39014/0 (socket says 172.21.15.73:39014) 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 -- 172.21.15.73:0/2734459211 learned_addr learned my addr 172.21.15.73:0/2734459211 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 -- 172.21.15.73:0/2734459211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf7e60 msgr2=0x55ce8fe02f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe02f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 -- 172.21.15.73:0/2734459211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 msgr2=0x55ce8fdfffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fdfffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 -- 172.21.15.73:0/2734459211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ce8fdf74c0 con 0x55ce8fe038b0 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f8419c93640 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8404009f30 tx=0x7f8404009f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5e7ecd7bb3cf415f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.350+0000 7f83fffff640 1 -- 172.21.15.73:0/2734459211 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840401a030 con 0x55ce8fe038b0 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f83fffff640 1 -- 172.21.15.73:0/2734459211 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404011e10 con 0x55ce8fe038b0 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f83fffff640 1 -- 172.21.15.73:0/2734459211 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8404010680 con 0x55ce8fe038b0 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- 172.21.15.73:0/2734459211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 msgr2=0x55ce8fe0a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 crypto rx=0x7f8404009f30 tx=0x7f8404009f60 comp rx=0 tx=0).stop 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- 172.21.15.73:0/2734459211 shutdown_connections 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fe038b0 0x55ce8fe0a910 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fdf8780 0x55ce8fdfffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 --2- 172.21.15.73:0/2734459211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe02f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- 172.21.15.73:0/2734459211 >> 172.21.15.73:0/2734459211 conn(0x55ce8fdf28d0 msgr2=0x55ce8fe1a410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- 172.21.15.73:0/2734459211 shutdown_connections 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- 172.21.15.73:0/2734459211 wait complete. 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 -- start start 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.351+0000 7f841f3bc740 1 Processor -- start 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe09030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe68b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141ab210 con 0x55ce8fdf7e60 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141ac530 con 0x55ce8fe038b0 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84141ab870 con 0x55ce8fdf8780 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 10 client.0 _ll_register_callbacks cb 0x7f84040095d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39026/0 (socket says 172.21.15.73:39026) 2024-03-09T00:07:14.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 -- 172.21.15.73:0/1863144126 learned_addr learned my addr 172.21.15.73:0/1863144126 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 -- 172.21.15.73:0/1863144126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 msgr2=0x55ce8fe09030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841a494640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe09030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe09030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 -- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 msgr2=0x55ce8fe68b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe68b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 -- 172.21.15.73:0/1863144126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ce8fe69100 con 0x55ce8fdf7e60 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841a494640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe09030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f8419c93640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8404009f00 tx=0x7f8404004a70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bbb2a2d32af8195c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f83fdffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f840403d060 con 0x55ce8fdf7e60 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8404011e10 con 0x55ce8fdf7e60 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8404010690 con 0x55ce8fdf7e60 2024-03-09T00:07:14.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f840401a050 con 0x55ce8fdf7e60 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 10 client.15711 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.352+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ce8fe6a400 con 0x55ce8fdf7e60 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8404011e10 con 0x55ce8fdf7e60 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f84040038f0 con 0x55ce8fdf7e60 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 10 client.15711 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 10 client.15711 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 20 client.15711 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe6a820 0x55ce8fe6cc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 4 client.15711 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ce8fd63750 con 0x55ce8fe6a820 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841a494640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe6a820 0x55ce8fe6cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f841a494640 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe6a820 0x55ce8fe6cc10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a6185edafb9833de server_cookie=133767bae819026b in_seq=0 out_seq=0 2024-03-09T00:07:14.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.353+0000 7f83fdffb640 10 client.15711 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f83fdffb640 1 -- 172.21.15.73:0/1863144126 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ce8fd63750 con 0x55ce8fe6a820 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f83fdffb640 10 client.15711 handle_command_reply: tid=0 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 client.15711 shutdown 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 msgr2=0x55ce8fe08af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 crypto rx=0x7f8404009f00 tx=0x7f8404004a70 comp rx=0 tx=0).stop 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 shutdown_connections 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ce8fe6a820 0x55ce8fe6cc10 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ce8fe038b0 0x55ce8fe68b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f83fdffb640 0 client.15711 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ce8fdf8780 0x55ce8fe09030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 --2- 172.21.15.73:0/1863144126 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ce8fdf7e60 0x55ce8fe08af0 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 >> 172.21.15.73:0/1863144126 conn(0x55ce8fdf28d0 msgr2=0x55ce8fdf1100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.355 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 shutdown_connections 2024-03-09T00:07:14.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 1 -- 172.21.15.73:0/1863144126 wait complete. 2024-03-09T00:07:14.356 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841f3bc740 20 client.15711 trim_cache size 0 max 16384 2024-03-09T00:07:14.356 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:07:14.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 msgr2=0x7f84000a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 0x7f84000a4f50 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f840c0cc810 tx=0x7f840c01a040 comp rx=0 tx=0).stop 2024-03-09T00:07:14.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 msgr2=0x7f8414196d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:14.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 crypto rx=0x7f840c00cad0 tx=0x7f840c00cf90 comp rx=0 tx=0).stop 2024-03-09T00:07:14.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 shutdown_connections 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84000a2a90 0x7f84000a4f50 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f84141968a0 0x7f8414196d50 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84141054b0 0x7f841419cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 --2- 172.21.15.73:0/1876642395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84141003e0 0x7f841419c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 >> 172.21.15.73:0/1876642395 conn(0x7f84140773a0 msgr2=0x7f84140fe310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 shutdown_connections 2024-03-09T00:07:14.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:14.354+0000 7f841c6f8640 1 -- 172.21.15.73:0/1876642395 wait complete. 2024-03-09T00:07:14.358 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:14 smithi073 ceph-mon[32309]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-09T00:07:14.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:14 smithi102 ceph-mon[36106]: pgmap v14: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 47 KiB/s wr, 24 op/s 2024-03-09T00:07:15.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/298839923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:15.361 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:07:15.361 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:07:15.361 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:07:15.390 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/298839923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:15.410 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/298839923' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 -- 172.21.15.73:0/4194393577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100dc0 msgr2=0x7f8da4104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 --2- 172.21.15.73:0/4194393577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100dc0 0x7f8da4104e00 secure :-1 s=READY pgs=540 cs=0 l=1 rev1=1 crypto rx=0x7f8d98009900 tx=0x7f8d9802bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 -- 172.21.15.73:0/4194393577 shutdown_connections 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 --2- 172.21.15.73:0/4194393577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41054e0 0x7f8da410d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 --2- 172.21.15.73:0/4194393577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100dc0 0x7f8da4104e00 unknown :-1 s=CLOSED pgs=540 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 --2- 172.21.15.73:0/4194393577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da4100410 0x7f8da41007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 -- 172.21.15.73:0/4194393577 >> 172.21.15.73:0/4194393577 conn(0x7f8da4076570 msgr2=0x7f8da4076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 -- 172.21.15.73:0/4194393577 shutdown_connections 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 -- 172.21.15.73:0/4194393577 wait complete. 2024-03-09T00:07:15.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.491+0000 7f8da89c4640 1 Processor -- start 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 -- start start 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da41054e0 0x7f8da4198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41994c0 0x7f8da41a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da4110550 con 0x7f8da4100410 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41103d0 con 0x7f8da41994c0 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da89c4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41106d0 con 0x7f8da41054e0 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39038/0 (socket says 172.21.15.73:39038) 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 -- 172.21.15.73:0/3973283626 learned_addr learned my addr 172.21.15.73:0/3973283626 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41994c0 0x7f8da41a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da41054e0 0x7f8da4198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 -- 172.21.15.73:0/3973283626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da41054e0 msgr2=0x7f8da4198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da41054e0 0x7f8da4198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 -- 172.21.15.73:0/3973283626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41994c0 msgr2=0x7f8da41a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41994c0 0x7f8da41a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.492+0000 7f8da259c640 1 -- 172.21.15.73:0/3973283626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8da41a4270 con 0x7f8da4100410 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8da259c640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d950 tx=0x7f8d9400de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8d937fe640 1 -- 172.21.15.73:0/3973283626 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94017070 con 0x7f8da4100410 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8d937fe640 1 -- 172.21.15.73:0/3973283626 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d94009c30 con 0x7f8da4100410 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8d937fe640 1 -- 172.21.15.73:0/3973283626 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9400f6b0 con 0x7f8da4100410 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8da41a4560 con 0x7f8da4100410 2024-03-09T00:07:15.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.493+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8da41a4ad0 con 0x7f8da4100410 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.495+0000 7f8d937fe640 1 -- 172.21.15.73:0/3973283626 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f8d940109d0 con 0x7f8da4100410 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 Processor -- start 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 -- start start 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8d937fe640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 0x7f8d8c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ed05280 0x55c66ed07670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 0x7f8d8c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ec5fb40 con 0x55c66ecfbd80 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ec08d80 con 0x55c66ed05280 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ec5bbe0 con 0x55c66ecfc6a0 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8d937fe640 1 -- 172.21.15.73:0/3973283626 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8d940cf0b0 con 0x7f8da4100410 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39050/0 (socket says 172.21.15.73:39050) 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da259c640 1 -- 172.21.15.73:0/1199337076 learned_addr learned my addr 172.21.15.73:0/1199337076 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.496+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ed05280 0x55c66ed07670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 -- 172.21.15.73:0/1199337076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 msgr2=0x55c66ed04d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 -- 172.21.15.73:0/1199337076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ed05280 msgr2=0x55c66ed07670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ed05280 0x55c66ed07670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 -- 172.21.15.73:0/1199337076 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c66ea3c7e0 con 0x55c66ecfbd80 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 0x7f8d8c0a4fd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c005fd0 tx=0x7f8d9c010740 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8da259c640 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8d94097b00 tx=0x7f8d94000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4598f5f2e4137d95 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8d90ff9640 1 -- 172.21.15.73:0/1199337076 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94020040 con 0x55c66ecfbd80 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8d90ff9640 1 -- 172.21.15.73:0/1199337076 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d940bb0c0 con 0x55c66ecfbd80 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.497+0000 7f8d90ff9640 1 -- 172.21.15.73:0/1199337076 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d940ccbc0 con 0x55c66ecfbd80 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- 172.21.15.73:0/1199337076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 msgr2=0x55c66ecfc160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f8d94097b00 tx=0x7f8d94000bc0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- 172.21.15.73:0/1199337076 shutdown_connections 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ed05280 0x55c66ed07670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/1199337076 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ecfc160 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- 172.21.15.73:0/1199337076 >> 172.21.15.73:0/1199337076 conn(0x55c66ecf64b0 msgr2=0x55c66ecf6890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- 172.21.15.73:0/1199337076 shutdown_connections 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- 172.21.15.73:0/1199337076 wait complete. 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- start start 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 Processor -- start 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed027a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ed08010 con 0x55c66ecfc6a0 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ec1c1c0 con 0x55c66ecfbd80 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.498+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c66ed08190 con 0x55c66ed05280 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8dab5a1740 10 client.0 _ll_register_callbacks cb 0x55c66ec0b7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39058/0 (socket says 172.21.15.73:39058) 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 -- 172.21.15.73:0/4146750463 learned_addr learned my addr 172.21.15.73:0/4146750463 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da259c640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed027a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 -- 172.21.15.73:0/4146750463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 msgr2=0x55c66ed027a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed027a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 -- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 msgr2=0x55c66ed04d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed04d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 -- 172.21.15.73:0/4146750463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c66ea9cf70 con 0x55c66ecfc6a0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000c00 tx=0x7f8d98005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dde096310fca32e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8d89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d980044d0 con 0x55c66ecfc6a0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8d98005030 con 0x55c66ecfc6a0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d940bb0c0 con 0x55c66ecfc6a0 2024-03-09T00:07:15.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.499+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d98005230 con 0x55c66ecfc6a0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 10 client.15723 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c66e934170 con 0x55c66ecfc6a0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d940bb0c0 con 0x55c66ecfc6a0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8d98003710 con 0x55c66ecfc6a0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 10 client.15723 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 10 client.15723 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 20 client.15723 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed153e0 0x55c66ed17800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 4 client.15723 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c66ec35960 con 0x55c66ed153e0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.500+0000 7f8da259c640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed153e0 0x55c66ed17800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8d89ffb640 10 client.15723 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8da259c640 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed153e0 0x55c66ed17800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b1ea8becf3697d32 server_cookie=493688e311d10c50 in_seq=0 out_seq=0 2024-03-09T00:07:15.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4146750463 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c66ec35960 con 0x55c66ed153e0 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8d89ffb640 10 client.15723 handle_command_reply: tid=0 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 client.15723 shutdown 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 msgr2=0x55c66ed020b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 crypto rx=0x7f8d98000c00 tx=0x7f8d98005d50 comp rx=0 tx=0).stop 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 shutdown_connections 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed153e0 0x55c66ed17800 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed027a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfc6a0 0x55c66ed020b0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4146750463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 >> 172.21.15.73:0/4146750463 conn(0x55c66ecf64b0 msgr2=0x55c66ecf8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8d89ffb640 0 client.15723 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 shutdown_connections 2024-03-09T00:07:15.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.501+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4146750463 wait complete. 2024-03-09T00:07:15.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.502+0000 7f8dab5a1740 20 client.15723 trim_cache size 0 max 16384 2024-03-09T00:07:15.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 Processor -- start 2024-03-09T00:07:15.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 -- start start 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed047e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41ac6e0 con 0x55c66ed05280 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41aeca0 con 0x55c66ecfbd80 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41acce0 con 0x55c66ecfc6a0 2024-03-09T00:07:15.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39064/0 (socket says 172.21.15.73:39064) 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 -- 172.21.15.73:0/694578634 learned_addr learned my addr 172.21.15.73:0/694578634 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da259c640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed047e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 -- 172.21.15.73:0/694578634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 msgr2=0x55c66ed04d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 -- 172.21.15.73:0/694578634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 msgr2=0x55c66ed047e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.517+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed047e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8da2d9d640 1 -- 172.21.15.73:0/694578634 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c66ed1ca30 con 0x55c66ed05280 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8da259c640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c011820 tx=0x7f8d9c015000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2c13f8c8f650f41 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8d8bfff640 1 -- 172.21.15.73:0/694578634 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c011e60 con 0x55c66ed05280 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8d8bfff640 1 -- 172.21.15.73:0/694578634 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d9c013be0 con 0x55c66ed05280 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8d8bfff640 1 -- 172.21.15.73:0/694578634 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d9c01e690 con 0x55c66ed05280 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 -- 172.21.15.73:0/694578634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 msgr2=0x55c66ed00980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c011820 tx=0x7f8d9c015000 comp rx=0 tx=0).stop 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 -- 172.21.15.73:0/694578634 shutdown_connections 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ed05280 0x55c66ed00980 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ecfc6a0 0x55c66ed04d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/694578634 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfbd80 0x55c66ed047e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 -- 172.21.15.73:0/694578634 >> 172.21.15.73:0/694578634 conn(0x55c66ecf64b0 msgr2=0x55c66ecfd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 -- 172.21.15.73:0/694578634 shutdown_connections 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.518+0000 7f8dab5a1740 1 -- 172.21.15.73:0/694578634 wait complete. 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 -- start start 2024-03-09T00:07:15.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 Processor -- start 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfc6a0 0x55c66ed09940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed6b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41ad320 con 0x55c66ecfbd80 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41ae640 con 0x55c66ecfc6a0 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8da41ad980 con 0x55c66ed05280 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8dab5a1740 10 client.0 _ll_register_callbacks cb 0x7f8d94096d30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39070/0 (socket says 172.21.15.73:39070) 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 -- 172.21.15.73:0/4294374870 learned_addr learned my addr 172.21.15.73:0/4294374870 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfc6a0 0x55c66ed09940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da2d9d640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed6b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 -- 172.21.15.73:0/4294374870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 msgr2=0x55c66ed6b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed6b920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 -- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfc6a0 msgr2=0x55c66ed09940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfc6a0 0x55c66ed09940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.519+0000 7f8da259c640 1 -- 172.21.15.73:0/4294374870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c66ed6be60 con 0x55c66ecfbd80 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8da259c640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d94097590 tx=0x7f8d94005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c50363f6af256354 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:15.527 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d94020040 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8d940a3440 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d9c013be0 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d940973d0 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 10 client.15729 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c66ed6d010 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d9c013be0 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8d940cc5d0 con 0x55c66ecfbd80 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 10 client.15729 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 10 client.15729 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 20 client.15729 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed6d300 0x55c66ed6f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 4 client.15729 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:07:15.528 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.520+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c66ec21080 con 0x55c66ed6d300 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.521+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed6d300 0x55c66ed6f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.521+0000 7f8d89ffb640 10 client.15729 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.521+0000 7f8da1d9b640 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed6d300 0x55c66ed6f6f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c33bc62e05b42c0c server_cookie=6638db9b24d17ce7 in_seq=0 out_seq=0 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.522+0000 7f8d89ffb640 1 -- 172.21.15.73:0/4294374870 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c66ec21080 con 0x55c66ed6d300 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.522+0000 7f8d89ffb640 10 client.15729 handle_command_reply: tid=0 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.522+0000 7f8dab5a1740 1 client.15729 shutdown 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.522+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 msgr2=0x55c66ed09400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.522+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 crypto rx=0x7f8d94097590 tx=0x7f8d94005f90 comp rx=0 tx=0).stop 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 shutdown_connections 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c66ed6d300 0x55c66ed6f6f0 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c66ed05280 0x55c66ed6b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.529 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c66ecfc6a0 0x55c66ed09940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8d89ffb640 0 client.15729 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 --2- 172.21.15.73:0/4294374870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c66ecfbd80 0x55c66ed09400 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 >> 172.21.15.73:0/4294374870 conn(0x55c66ecf64b0 msgr2=0x55c66ed008b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 shutdown_connections 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 1 -- 172.21.15.73:0/4294374870 wait complete. 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8dab5a1740 20 client.15729 trim_cache size 0 max 16384 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:15.530 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:07:15.531 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "911fdb97-a028-4169-b2c5-f592746342fd", 2024-03-09T00:07:15.531 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:07:15.531 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:15.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 msgr2=0x7f8d8c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 0x7f8d8c0a4fd0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f8d9c005fd0 tx=0x7f8d9c010740 comp rx=0 tx=0).stop 2024-03-09T00:07:15.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 msgr2=0x7f8da41989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 crypto rx=0x7f8d9400d950 tx=0x7f8d9400de10 comp rx=0 tx=0).stop 2024-03-09T00:07:15.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 shutdown_connections 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8da41994c0 0x7f8da41a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8da41054e0 0x7f8da4198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8d8c0a2b10 0x7f8d8c0a4fd0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 --2- 172.21.15.73:0/3973283626 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8da4100410 0x7f8da41989f0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 >> 172.21.15.73:0/3973283626 conn(0x7f8da4076570 msgr2=0x7f8da40fe170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 shutdown_connections 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.523+0000 7f8da89c4640 1 -- 172.21.15.73:0/3973283626 wait complete. 2024-03-09T00:07:15.533 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:07:15.534 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 -- 172.21.15.73:0/3375229297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 msgr2=0x7f6f8410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/3375229297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8410d070 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c009ca0 tx=0x7f6f7c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 -- 172.21.15.73:0/3375229297 shutdown_connections 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/3375229297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8410d070 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/3375229297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84100d80 0x7f6f84104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/3375229297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 0x7f6f841007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 -- 172.21.15.73:0/3375229297 >> 172.21.15.73:0/3375229297 conn(0x7f6f840772f0 msgr2=0x7f6f84077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 -- 172.21.15.73:0/3375229297 shutdown_connections 2024-03-09T00:07:15.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.683+0000 7f6f8c10a640 1 -- 172.21.15.73:0/3375229297 wait complete. 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 Processor -- start 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 -- start start 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 0x7f6f8419d780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 0x7f6f84197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f84111670 con 0x7f6f841054a0 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841114f0 con 0x7f6f841003d0 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8c10a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841117f0 con 0x7f6f84197930 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39098/0 (socket says 172.21.15.73:39098) 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 -- 172.21.15.73:0/2999944294 learned_addr learned my addr 172.21.15.73:0/2999944294 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 0x7f6f8419d780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 0x7f6f84197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 -- 172.21.15.73:0/2999944294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 msgr2=0x7f6f84197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 0x7f6f84197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 -- 172.21.15.73:0/2999944294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 msgr2=0x7f6f8419d780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 0x7f6f8419d780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 -- 172.21.15.73:0/2999944294 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f84198450 con 0x7f6f841054a0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 0x7f6f84197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.684+0000 7f6f896a5640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400e9f0 tx=0x7f6f7400eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.685+0000 7f6f7affd640 1 -- 172.21.15.73:0/2999944294 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7400ccc0 con 0x7f6f841054a0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.685+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f84198740 con 0x7f6f841054a0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.685+0000 7f6f7affd640 1 -- 172.21.15.73:0/2999944294 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f74005500 con 0x7f6f841054a0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.685+0000 7f6f7affd640 1 -- 172.21.15.73:0/2999944294 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f74010690 con 0x7f6f841054a0 2024-03-09T00:07:15.682 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.685+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f841a4940 con 0x7f6f841054a0 2024-03-09T00:07:15.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f7affd640 1 -- 172.21.15.73:0/2999944294 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f6f740046c0 con 0x7f6f841054a0 2024-03-09T00:07:15.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 Processor -- start 2024-03-09T00:07:15.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 -- start start 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f7affd640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 0x7f6f700a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a0b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb303e00 con 0x55bdeb3a9c60 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb2acb10 con 0x55bdeb3a1080 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb2ffea0 con 0x55bdeb3a0760 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f7affd640 1 -- 172.21.15.73:0/2999944294 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6f740cf840 con 0x7f6f841054a0 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.687+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 0x7f6f700a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39108/0 (socket says 172.21.15.73:39108) 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 -- 172.21.15.73:0/180884550 learned_addr learned my addr 172.21.15.73:0/180884550 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f896a5640 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a0b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 -- 172.21.15.73:0/180884550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a0760 msgr2=0x55bdeb3a0b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a0b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 -- 172.21.15.73:0/180884550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 msgr2=0x55bdeb3a9720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 -- 172.21.15.73:0/180884550 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdeb0e27d0 con 0x55bdeb3a9c60 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 0x7f6f700a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c036000 tx=0x7f6f7c0324f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0326a0 tx=0x7f6f7c045000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c6993f47d3ba371 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f6ffff640 1 -- 172.21.15.73:0/180884550 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c050070 con 0x55bdeb3a9c60 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f6ffff640 1 -- 172.21.15.73:0/180884550 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f7c03a040 con 0x55bdeb3a9c60 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.688+0000 7f6f6ffff640 1 -- 172.21.15.73:0/180884550 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f7c039070 con 0x55bdeb3a9c60 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- 172.21.15.73:0/180884550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 msgr2=0x55bdeb3ac050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c0326a0 tx=0x7f6f7c045000 comp rx=0 tx=0).stop 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- 172.21.15.73:0/180884550 shutdown_connections 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3ac050 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/180884550 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a0b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- 172.21.15.73:0/180884550 >> 172.21.15.73:0/180884550 conn(0x55bdeb39ae90 msgr2=0x55bdeb39b270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- 172.21.15.73:0/180884550 shutdown_connections 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- 172.21.15.73:0/180884550 wait complete. 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 -- start start 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.689+0000 7f6f8edbe740 1 Processor -- start 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb3ac9f0 con 0x55bdeb3a1080 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb2c00a0 con 0x55bdeb3a0760 2024-03-09T00:07:15.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bdeb3acb70 con 0x55bdeb3a9c60 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8edbe740 10 client.0 _ll_register_callbacks cb 0x55bdeb2bfa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39116/0 (socket says 172.21.15.73:39116) 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 -- 172.21.15.73:0/272301738 learned_addr learned my addr 172.21.15.73:0/272301738 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 -- 172.21.15.73:0/272301738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 msgr2=0x55bdeb3a6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a6c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 -- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a0760 msgr2=0x55bdeb3a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a9380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 -- 172.21.15.73:0/272301738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdeb0183f0 con 0x55bdeb3a1080 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f896a5640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400cf10 tx=0x7f6f7409b2e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95933b9a14116d7c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.690+0000 7f6f6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f740b3020 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6f740cf430 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f7c03a040 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f740862c0 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 10 client.15741 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55bdeafd80f0 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f7c03a040 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6f740ccd60 con 0x55bdeb3a1080 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 10 client.15741 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 10 client.15741 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 20 client.15741 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb3b95c0 0x55bdeb3bb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 4 client.15741 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:15.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bdeb311340 con 0x55bdeb3b95c0 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.691+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb3b95c0 0x55bdeb3bb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f6d7fa640 10 client.15741 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb3b95c0 0x55bdeb3bb9b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d28dcebb6dcadb9 server_cookie=7493ce88556b4cc8 in_seq=0 out_seq=0 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/272301738 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55bdeb311340 con 0x55bdeb3b95c0 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f6d7fa640 10 client.15741 handle_command_reply: tid=0 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 client.15741 shutdown 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 msgr2=0x55bdeb3a6710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400cf10 tx=0x7f6f7409b2e0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 shutdown_connections 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb3b95c0 0x55bdeb3bb9b0 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a6710 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/272301738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 >> 172.21.15.73:0/272301738 conn(0x55bdeb39ae90 msgr2=0x55bdeb39cbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.692+0000 7f6f6d7fa640 0 client.15741 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.693+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 shutdown_connections 2024-03-09T00:07:15.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.693+0000 7f6f8edbe740 1 -- 172.21.15.73:0/272301738 wait complete. 2024-03-09T00:07:15.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.693+0000 7f6f8edbe740 20 client.15741 trim_cache size 0 max 16384 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 Processor -- start 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 -- start start 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a3970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.707+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841ac660 con 0x55bdeb3a0760 2024-03-09T00:07:15.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841aec20 con 0x55bdeb3a9c60 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841acc60 con 0x55bdeb3a1080 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39132/0 (socket says 172.21.15.73:39132) 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1498365734 learned_addr learned my addr 172.21.15.73:0/1498365734 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f8a6a7640 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a3970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1498365734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a1080 msgr2=0x55bdeb3a9380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1498365734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a9c60 msgr2=0x55bdeb3a3970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a3970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1498365734 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bdeb3c0b80 con 0x55bdeb3a0760 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009730 tx=0x7f6f80005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=947735176aa2c320 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f6f7fe640 1 -- 172.21.15.73:0/1498365734 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80004310 con 0x55bdeb3a0760 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f6f7fe640 1 -- 172.21.15.73:0/1498365734 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f800044b0 con 0x55bdeb3a0760 2024-03-09T00:07:15.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.708+0000 7f6f6f7fe640 1 -- 172.21.15.73:0/1498365734 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f8000f3c0 con 0x55bdeb3a0760 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1498365734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 msgr2=0x55bdeb3a8e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 crypto rx=0x7f6f80009730 tx=0x7f6f80005f90 comp rx=0 tx=0).stop 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1498365734 shutdown_connections 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3a3970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a1080 0x55bdeb3a9380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1498365734 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3a8e40 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1498365734 >> 172.21.15.73:0/1498365734 conn(0x55bdeb39ae90 msgr2=0x55bdeb3a66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1498365734 shutdown_connections 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1498365734 wait complete. 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 -- start start 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 Processor -- start 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3afcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.709+0000 7f6f8edbe740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3b01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841ad2a0 con 0x55bdeb3a0760 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841ae5c0 con 0x55bdeb3a1080 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f841ad900 con 0x55bdeb3a9c60 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 10 client.0 _ll_register_callbacks cb 0x7f6f74009750 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39142/0 (socket says 172.21.15.73:39142) 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1918581036 learned_addr learned my addr 172.21.15.73:0/1918581036 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f896a5640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3afcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1918581036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 msgr2=0x55bdeb3b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3b01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 msgr2=0x55bdeb3afcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:15.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3afcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 -- 172.21.15.73:0/1918581036 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bdeb40fa80 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f89ea6640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6f80000c00 tx=0x7f6f80005730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=137e820edf1f9afc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f6d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6f80003e10 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6f800035f0 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6f800044b0 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6f80039050 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 10 client.15747 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55bdeb410c70 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.710+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6f800044b0 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6f8001ece0 con 0x55bdeb3a0760 2024-03-09T00:07:15.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 10 client.15747 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 10 client.15747 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 20 client.15747 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb4112b0 0x55bdeb4136d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 4 client.15747 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55bdeb30b690 con 0x55bdeb4112b0 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f896a5640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb4112b0 0x55bdeb4136d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f896a5640 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb4112b0 0x55bdeb4136d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c3be9a411e17823 server_cookie=bd829b739d501b1d in_seq=0 out_seq=0 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f6d7fa640 10 client.15747 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f6d7fa640 1 -- 172.21.15.73:0/1918581036 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55bdeb30b690 con 0x55bdeb4112b0 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f6d7fa640 10 client.15747 handle_command_reply: tid=0 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 1 client.15747 shutdown 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 msgr2=0x55bdeb3af770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.711+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 crypto rx=0x7f6f80000c00 tx=0x7f6f80005730 comp rx=0 tx=0).stop 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 shutdown_connections 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55bdeb4112b0 0x55bdeb4136d0 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55bdeb3a9c60 0x55bdeb3b01f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55bdeb3a1080 0x55bdeb3afcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 --2- 172.21.15.73:0/1918581036 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55bdeb3a0760 0x55bdeb3af770 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f6d7fa640 0 client.15747 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 >> 172.21.15.73:0/1918581036 conn(0x55bdeb39ae90 msgr2=0x55bdeb3a1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 shutdown_connections 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 1 -- 172.21.15.73:0/1918581036 wait complete. 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8edbe740 20 client.15747 trim_cache size 0 max 16384 2024-03-09T00:07:15.714 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "911fdb97-a028-4169-b2c5-f592746342fd": { 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "911fdb97-a028-4169-b2c5-f592746342fd", 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:07:15.715 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 msgr2=0x7f6f700a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 0x7f6f700a4f50 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f6f7c036000 tx=0x7f6f7c0324f0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 msgr2=0x7f6f8419dcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 crypto rx=0x7f6f7400e9f0 tx=0x7f6f7400eeb0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 shutdown_connections 2024-03-09T00:07:15.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6f84197930 0x7f6f84197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6f700a2a90 0x7f6f700a4f50 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6f841054a0 0x7f6f8419dcc0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 --2- 172.21.15.73:0/2999944294 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6f841003d0 0x7f6f8419d780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 >> 172.21.15.73:0/2999944294 conn(0x7f6f840772f0 msgr2=0x7f6f840fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 shutdown_connections 2024-03-09T00:07:15.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:15.712+0000 7f6f8c10a640 1 -- 172.21.15.73:0/2999944294 wait complete. 2024-03-09T00:07:15.718 INFO:tasks.cephfs.filesystem:scrub status for tag:911fdb97-a028-4169-b2c5-f592746342fd - {'path': '/', 'tag': '911fdb97-a028-4169-b2c5-f592746342fd', 'options': 'recursive,force'} 2024-03-09T00:07:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:16 smithi179 ceph-mon[36637]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-09T00:07:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:16 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:07:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:16 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:16 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:07:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:16 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:16 smithi073 ceph-mon[32309]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-09T00:07:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:16 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:07:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:16 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:16 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:07:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:16 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:16 smithi102 ceph-mon[36106]: pgmap v15: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 2.7 KiB/s wr, 6 op/s 2024-03-09T00:07:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:16 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:07:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:16 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:16 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:07:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:16 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:18 smithi179 ceph-mon[36637]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-09T00:07:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:18 smithi073 ceph-mon[32309]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-09T00:07:18.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:18 smithi102 ceph-mon[36106]: pgmap v16: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 2.7 KiB/s wr, 8 op/s 2024-03-09T00:07:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:19 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:07:19.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:19 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 -- 172.21.15.73:0/3146972654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 msgr2=0x7f0e0c0ffde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/3146972654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c0ffde0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 crypto rx=0x7f0df4009900 tx=0x7f0df402bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 -- 172.21.15.73:0/3146972654 shutdown_connections 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/3146972654 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 0x7f0e0c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/3146972654 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1003b0 0x7f0e0c105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.489 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/3146972654 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c0ffde0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 -- 172.21.15.73:0/3146972654 >> 172.21.15.73:0/3146972654 conn(0x7f0e0c076710 msgr2=0x7f0e0c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 -- 172.21.15.73:0/3146972654 shutdown_connections 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.492+0000 7f0e10a8c640 1 -- 172.21.15.73:0/3146972654 wait complete. 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 Processor -- start 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 -- start start 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 0x7f0e0c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1979d0 0x7f0e0c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1116d0 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c111550 con 0x7f0e0c106670 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e10a8c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c111850 con 0x7f0e0c1979d0 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.490 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59290/0 (socket says 172.21.15.73:59290) 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e0a59c640 1 -- 172.21.15.73:0/1783590372 learned_addr learned my addr 172.21.15.73:0/1783590372 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.493+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1979d0 0x7f0e0c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 0x7f0e0c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 -- 172.21.15.73:0/1783590372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1979d0 msgr2=0x7f0e0c197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1979d0 0x7f0e0c197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 -- 172.21.15.73:0/1783590372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 msgr2=0x7f0e0c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 0x7f0e0c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 -- 172.21.15.73:0/1783590372 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e0c1984f0 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0df4006010 tx=0x7f0df402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e037fe640 1 -- 172.21.15.73:0/1783590372 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4041070 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e0c198780 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e037fe640 1 -- 172.21.15.73:0/1783590372 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df4004450 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e037fe640 1 -- 172.21.15.73:0/1783590372 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df4034440 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.491 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.494+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e0c1a4970 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.496+0000 7f0e037fe640 1 -- 172.21.15.73:0/1783590372 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f0df4044070 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.496+0000 7f0e13674740 1 Processor -- start 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 -- start start 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e037fe640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 0x7f0df80a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x5560217017e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021715140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.498 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560216655f0 con 0x556021700a40 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55602160e930 con 0x556021701360 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e13674740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556021661690 con 0x556021701e00 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 0x7f0df80a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59302/0 (socket says 172.21.15.73:59302) 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0a59c640 1 -- 172.21.15.73:0/2959528577 learned_addr learned my addr 172.21.15.73:0/2959528577 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e037fe640 1 -- 172.21.15.73:0/1783590372 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0df40f10d0 con 0x7f0e0c0ffa00 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021715140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x5560217017e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0a59c640 1 -- 172.21.15.73:0/2959528577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 msgr2=0x556021715140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.497+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021715140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e0a59c640 1 -- 172.21.15.73:0/2959528577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 msgr2=0x5560217017e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x5560217017e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.499 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e0a59c640 1 -- 172.21.15.73:0/2959528577 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55602137a210 con 0x556021700a40 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x5560217017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0df40ee530 tx=0x7f0df4046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a2c5933a27fa44 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 0x7f0df80a4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005cc0 tx=0x7f0e04012960 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e00ff9640 1 -- 172.21.15.73:0/2959528577 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df404b040 con 0x556021700a40 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e00ff9640 1 -- 172.21.15.73:0/2959528577 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df40bd960 con 0x556021700a40 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.498+0000 7f0e00ff9640 1 -- 172.21.15.73:0/2959528577 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df40c13b0 con 0x556021700a40 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- 172.21.15.73:0/2959528577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 msgr2=0x556021700e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 crypto rx=0x7f0df40ee530 tx=0x7f0df4046000 comp rx=0 tx=0).stop 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- 172.21.15.73:0/2959528577 shutdown_connections 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021715140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x5560217017e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- 172.21.15.73:0/2959528577 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021700e20 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- 172.21.15.73:0/2959528577 >> 172.21.15.73:0/2959528577 conn(0x5560216c85b0 msgr2=0x5560216fbcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.500 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- 172.21.15.73:0/2959528577 shutdown_connections 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- 172.21.15.73:0/2959528577 wait complete. 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- start start 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 Processor -- start 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021700a40 0x556021712700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701e00 0x556021713180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556021715870 con 0x556021701360 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556021621d90 con 0x556021701e00 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5560217159f0 con 0x556021700a40 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.499+0000 7f0e13674740 10 client.0 _ll_register_callbacks cb 0x556021379a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59310/0 (socket says 172.21.15.73:59310) 2024-03-09T00:07:19.501 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 -- 172.21.15.73:0/3131637954 learned_addr learned my addr 172.21.15.73:0/3131637954 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701e00 0x556021713180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 -- 172.21.15.73:0/3131637954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021700a40 msgr2=0x556021712700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021700a40 0x556021712700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 -- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701e00 msgr2=0x556021713180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701e00 0x556021713180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 -- 172.21.15.73:0/3131637954 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556021446620 con 0x556021701360 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00be80 tx=0x7f0dfc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=394db816d5abb9ba server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc005220 con 0x556021701360 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0dfc0053c0 con 0x556021701360 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0df40bd960 con 0x556021701360 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dfc024050 con 0x556021701360 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e13674740 10 client.15759 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:19.502 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556021721b40 con 0x556021701360 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.500+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df40bd960 con 0x556021701360 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0dfc01b7e0 con 0x556021701360 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 10 client.15759 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 10 client.15759 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 20 client.15759 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556021723000 0x556021725450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 4 client.15759 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556021672b30 con 0x556021723000 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556021723000 0x556021725450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556021723000 0x556021725450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c2dd36a986f3c7d server_cookie=56244df37a6381d8 in_seq=0 out_seq=0 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.501+0000 7f0df1ffb640 10 client.15759 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0df1ffb640 1 -- 172.21.15.73:0/3131637954 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556021672b30 con 0x556021723000 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0df1ffb640 10 client.15759 handle_command_reply: tid=0 2024-03-09T00:07:19.503 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 client.15759 shutdown 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 msgr2=0x556021712c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc00be80 tx=0x7f0dfc0117b0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 shutdown_connections 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556021723000 0x556021725450 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701e00 0x556021713180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x556021712c40 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0df1ffb640 0 client.15759 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 --2- 172.21.15.73:0/3131637954 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021700a40 0x556021712700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 >> 172.21.15.73:0/3131637954 conn(0x5560216c85b0 msgr2=0x5560216fda10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 shutdown_connections 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 1 -- 172.21.15.73:0/3131637954 wait complete. 2024-03-09T00:07:19.504 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.502+0000 7f0e13674740 20 client.15759 trim_cache size 0 max 16384 2024-03-09T00:07:19.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:19 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:07:19.515 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.515+0000 7f0e13674740 1 Processor -- start 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.515+0000 7f0e13674740 1 -- start start 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x556021710120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021719c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1aa580 con 0x556021700a40 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1acb40 con 0x556021701360 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e13674740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1aab80 con 0x556021701e00 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.516 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59316/0 (socket says 172.21.15.73:59316) 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 -- 172.21.15.73:0/829746588 learned_addr learned my addr 172.21.15.73:0/829746588 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x556021710120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 -- 172.21.15.73:0/829746588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 msgr2=0x556021719c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021719c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 -- 172.21.15.73:0/829746588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 msgr2=0x556021710120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x556021710120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 -- 172.21.15.73:0/829746588 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556021704a00 con 0x556021700a40 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x556021710120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0df40d3180 tx=0x7f0df40bd280 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86873f87189cc78f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0df3fff640 1 -- 172.21.15.73:0/829746588 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df404b040 con 0x556021700a40 2024-03-09T00:07:19.517 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0df3fff640 1 -- 172.21.15.73:0/829746588 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df40cf9f0 con 0x556021700a40 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.516+0000 7f0df3fff640 1 -- 172.21.15.73:0/829746588 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0df40c13b0 con 0x556021700a40 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- 172.21.15.73:0/829746588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 msgr2=0x556021714e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 crypto rx=0x7f0df40d3180 tx=0x7f0df40bd280 comp rx=0 tx=0).stop 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- 172.21.15.73:0/829746588 shutdown_connections 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021719c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021701360 0x556021710120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 --2- 172.21.15.73:0/829746588 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021700a40 0x556021714e30 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- 172.21.15.73:0/829746588 >> 172.21.15.73:0/829746588 conn(0x5560216c85b0 msgr2=0x5560216fb7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- 172.21.15.73:0/829746588 shutdown_connections 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- 172.21.15.73:0/829746588 wait complete. 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 -- start start 2024-03-09T00:07:19.518 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.517+0000 7f0e13674740 1 Processor -- start 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021700a40 0x556021719c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021717c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1ab1c0 con 0x556021701360 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1ac4e0 con 0x556021700a40 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e0c1ab820 con 0x556021701e00 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 10 client.0 _ll_register_callbacks cb 0x7f0df4003ef0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:59330/0 (socket says 172.21.15.73:59330) 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 -- 172.21.15.73:0/1405122060 learned_addr learned my addr 172.21.15.73:0/1405122060 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021700a40 0x556021719c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.519 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 -- 172.21.15.73:0/1405122060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 msgr2=0x556021717c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021717c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021717c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 -- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021700a40 msgr2=0x556021719c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021700a40 0x556021719c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 -- 172.21.15.73:0/1405122060 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556021718480 con 0x556021701360 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e0ad9d640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021717c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e09d9b640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0058c0 tx=0x7f0dfc00b620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f7577129be3303fd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0df1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0dfc021040 con 0x556021701360 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0dfc01b890 con 0x556021701360 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0df40cf9f0 con 0x556021701360 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0dfc039050 con 0x556021701360 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 10 client.15765 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:19.520 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.518+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55602177b090 con 0x556021701360 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0df40cf9f0 con 0x556021701360 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0dfc004b40 con 0x556021701360 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 10 client.15765 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 10 client.15765 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 20 client.15765 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55602177b7f0 0x55602177dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 4 client.15765 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55602166ce80 con 0x55602177b7f0 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.519+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55602177b7f0 0x55602177dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.520+0000 7f0df1ffb640 10 client.15765 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:19.521 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.520+0000 7f0e0a59c640 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55602177b7f0 0x55602177dc10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a8fde510c59e62cd server_cookie=b4bc5dc72272b71a in_seq=0 out_seq=0 2024-03-09T00:07:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0df1ffb640 1 -- 172.21.15.73:0/1405122060 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55602166ce80 con 0x55602177b7f0 2024-03-09T00:07:19.522 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0df1ffb640 10 client.15765 handle_command_reply: tid=0 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 client.15765 shutdown 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 msgr2=0x55602171a170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 crypto rx=0x7f0dfc0058c0 tx=0x7f0dfc00b620 comp rx=0 tx=0).stop 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 shutdown_connections 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55602177b7f0 0x55602177dc10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556021701e00 0x556021717c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556021701360 0x55602171a170 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0df1ffb640 0 client.15765 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 --2- 172.21.15.73:0/1405122060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556021700a40 0x556021719c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 >> 172.21.15.73:0/1405122060 conn(0x5560216c85b0 msgr2=0x556021710f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 shutdown_connections 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 1 -- 172.21.15.73:0/1405122060 wait complete. 2024-03-09T00:07:19.523 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e13674740 20 client.15765 trim_cache size 0 max 16384 2024-03-09T00:07:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:07:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:07:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:07:19.524 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 msgr2=0x7f0df80a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 0x7f0df80a4ff0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f0e04005cc0 tx=0x7f0e04012960 comp rx=0 tx=0).stop 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 msgr2=0x7f0e0c19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 crypto rx=0x7f0df4006010 tx=0x7f0df402cc70 comp rx=0 tx=0).stop 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 shutdown_connections 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0e0c1979d0 0x7f0e0c197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0e0c106670 0x7f0e0c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0df80a2b30 0x7f0df80a4ff0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.525 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 --2- 172.21.15.73:0/1783590372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0e0c0ffa00 0x7f0e0c19d820 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 >> 172.21.15.73:0/1783590372 conn(0x7f0e0c076710 msgr2=0x7f0e0c1047e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.526+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 shutdown_connections 2024-03-09T00:07:19.526 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:19.527+0000 7f0e10a8c640 1 -- 172.21.15.73:0/1783590372 wait complete. 2024-03-09T00:07:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:20 smithi179 ceph-mon[36637]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-09T00:07:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:20 smithi073 ceph-mon[32309]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-09T00:07:20.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:20 smithi102 ceph-mon[36106]: pgmap v17: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 682 B/s wr, 4 op/s 2024-03-09T00:07:22.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:22 smithi179 ceph-mon[36637]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-09T00:07:22.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:22 smithi073 ceph-mon[32309]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-09T00:07:22.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:22 smithi102 ceph-mon[36106]: pgmap v18: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 2 op/s 2024-03-09T00:07:24.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:24 smithi179 ceph-mon[36637]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-09T00:07:24.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:24 smithi073 ceph-mon[32309]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-09T00:07:24.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:24 smithi102 ceph-mon[36106]: pgmap v19: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 511 B/s wr, 4 op/s 2024-03-09T00:07:26.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:26 smithi179 ceph-mon[36637]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-09T00:07:26.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:26 smithi073 ceph-mon[32309]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-09T00:07:26.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:26 smithi102 ceph-mon[36106]: pgmap v20: 129 pgs: 129 active+clean; 1.3 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-09T00:07:28.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:28 smithi179 ceph-mon[36637]: pgmap v21: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-09T00:07:28.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:28 smithi073 ceph-mon[32309]: pgmap v21: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-09T00:07:28.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:28 smithi102 ceph-mon[36106]: pgmap v21: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.8 KiB/s wr, 4 op/s 2024-03-09T00:07:30.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:30 smithi179 ceph-mon[36637]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:30.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:30 smithi073 ceph-mon[32309]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:30.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:30 smithi102 ceph-mon[36106]: pgmap v22: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:32.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:32 smithi179 ceph-mon[36637]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:32.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:32 smithi073 ceph-mon[32309]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:32 smithi102 ceph-mon[36106]: pgmap v23: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 1.8 KiB/s wr, 2 op/s 2024-03-09T00:07:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 -- 172.21.15.73:0/3695010592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100440 msgr2=0x7f15e81008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/3695010592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100440 0x7f15e81008c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 crypto rx=0x7f15d0009900 tx=0x7f15d002bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 -- 172.21.15.73:0/3695010592 shutdown_connections 2024-03-09T00:07:32.611 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/3695010592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100e00 0x7f15e810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/3695010592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100440 0x7f15e81008c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/3695010592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 0x7f15e8105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 -- 172.21.15.73:0/3695010592 >> 172.21.15.73:0/3695010592 conn(0x7f15e8077040 msgr2=0x7f15e8077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 -- 172.21.15.73:0/3695010592 shutdown_connections 2024-03-09T00:07:32.612 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.616+0000 7f15ed1a9640 1 -- 172.21.15.73:0/3695010592 wait complete. 2024-03-09T00:07:32.614 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 Processor -- start 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 -- start start 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 0x7f15e819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 0x7f15e8196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e8110650 con 0x7f15e8100e00 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81104d0 con 0x7f15e8105190 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15ed1a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81107d0 con 0x7f15e8100440 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.617+0000 7f15dffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50384/0 (socket says 172.21.15.73:50384) 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 -- 172.21.15.73:0/1225272406 learned_addr learned my addr 172.21.15.73:0/1225272406 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:32.615 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e759e640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 0x7f15e8196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 0x7f15e819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 -- 172.21.15.73:0/1225272406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 msgr2=0x7f15e819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 0x7f15e819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 -- 172.21.15.73:0/1225272406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 msgr2=0x7f15e8196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 0x7f15e8196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 -- 172.21.15.73:0/1225272406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15e8196ff0 con 0x7f15e8100e00 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 0x7f15e819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15dffff640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005940 tx=0x7f15d00043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e4d99640 1 -- 172.21.15.73:0/1225272406 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d002ccc0 con 0x7f15e8100e00 2024-03-09T00:07:32.616 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e4d99640 1 -- 172.21.15.73:0/1225272406 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d00054e0 con 0x7f15e8100e00 2024-03-09T00:07:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15e8197280 con 0x7f15e8100e00 2024-03-09T00:07:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15e4d99640 1 -- 172.21.15.73:0/1225272406 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d003b440 con 0x7f15e8100e00 2024-03-09T00:07:32.617 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.618+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15e8068970 con 0x7f15e8100e00 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.620+0000 7f15e4d99640 1 -- 172.21.15.73:0/1225272406 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f15d0003680 con 0x7f15e8100e00 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 Processor -- start 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 -- start start 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f560664a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f560670940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5605c95f0 con 0x55f560664fd0 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f560572930 con 0x55f560665d40 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15efd83740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f5605c5690 con 0x55f56066e550 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.621+0000 7f15e4d99640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 0x7f15d40a4ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.624 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50392/0 (socket says 172.21.15.73:50392) 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 -- 172.21.15.73:0/3399851252 learned_addr learned my addr 172.21.15.73:0/3399851252 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e759e640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 0x7f15d40a4ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e759e640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f560664a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e4d99640 1 -- 172.21.15.73:0/1225272406 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15d00f3120 con 0x7f15e8100e00 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15dffff640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f560670940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 -- 172.21.15.73:0/3399851252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 msgr2=0x55f560670940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f560670940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 -- 172.21.15.73:0/3399851252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 msgr2=0x55f560664a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f560664a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 -- 172.21.15.73:0/3399851252 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5602de210 con 0x55f560664fd0 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15dffff640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f560670940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:32.625 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e759e640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 0x7f15d40a4ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005fd0 tx=0x7f15e0009ad0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009c30 tx=0x7f15d800e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1670e47a41dd9a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15dd7fa640 1 -- 172.21.15.73:0/3399851252 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8016070 con 0x55f560664fd0 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15dd7fa640 1 -- 172.21.15.73:0/3399851252 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d8005500 con 0x55f560664fd0 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.622+0000 7f15dd7fa640 1 -- 172.21.15.73:0/3399851252 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8011470 con 0x55f560664fd0 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- 172.21.15.73:0/3399851252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 msgr2=0x55f560665450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 crypto rx=0x7f15d8009c30 tx=0x7f15d800e6c0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- 172.21.15.73:0/3399851252 shutdown_connections 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f560670940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f560665450 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 --2- 172.21.15.73:0/3399851252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f560664a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- 172.21.15.73:0/3399851252 >> 172.21.15.73:0/3399851252 conn(0x55f56062c5b0 msgr2=0x55f56065fcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- 172.21.15.73:0/3399851252 shutdown_connections 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- 172.21.15.73:0/3399851252 wait complete. 2024-03-09T00:07:32.626 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.623+0000 7f15efd83740 1 -- start start 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 Processor -- start 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f56066ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f56066bfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f560671280 con 0x55f560664fd0 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f560585d90 con 0x55f560665d40 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f560671400 con 0x55f56066e550 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15efd83740 10 client.0 _ll_register_callbacks cb 0x55f5602dda00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50408/0 (socket says 172.21.15.73:50408) 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 -- 172.21.15.73:0/1321525452 learned_addr learned my addr 172.21.15.73:0/1321525452 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15dffff640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f56066ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.627 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 -- 172.21.15.73:0/1321525452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 msgr2=0x55f56066bfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f56066bfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 -- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 msgr2=0x55f56066ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f56066ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 -- 172.21.15.73:0/1321525452 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5603aa620 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15dffff640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f56066ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005e70 tx=0x7f15d80048b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81aec34c6a76a888 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.624+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d8028060 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15d8004ea0 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15d8005500 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15d8016040 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 10 client.15777 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f56067d490 con 0x55f560664fd0 2024-03-09T00:07:32.628 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15d8005500 con 0x55f560664fd0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f15d800f040 con 0x55f560664fd0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 10 client.15777 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 10 client.15777 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 20 client.15777 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f56067e8f0 0x55f560680ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 4 client.15777 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.625+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f5605d6b30 con 0x55f56067e8f0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.626+0000 7f15dffff640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f56067e8f0 0x55f560680ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.626+0000 7f15ce7fc640 10 client.15777 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.626+0000 7f15dffff640 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f56067e8f0 0x55f560680ce0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9aea3d97253cbebe server_cookie=375d84c85bfbb78c in_seq=0 out_seq=0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1321525452 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f5605d6b30 con 0x55f56067e8f0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15ce7fc640 10 client.15777 handle_command_reply: tid=0 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 client.15777 shutdown 2024-03-09T00:07:32.629 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 msgr2=0x55f56066b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 crypto rx=0x7f15d8005e70 tx=0x7f15d80048b0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 shutdown_connections 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f56067e8f0 0x55f560680ce0 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f56066e550 0x55f56066bfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560665d40 0x55f56066ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15ce7fc640 0 client.15777 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 --2- 172.21.15.73:0/1321525452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560664fd0 0x55f56066b550 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 >> 172.21.15.73:0/1321525452 conn(0x55f56062c5b0 msgr2=0x55f560661a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 shutdown_connections 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 1 -- 172.21.15.73:0/1321525452 wait complete. 2024-03-09T00:07:32.630 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.627+0000 7f15efd83740 20 client.15777 trim_cache size 0 max 16384 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.642+0000 7f15efd83740 1 Processor -- start 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.642+0000 7f15efd83740 1 -- start start 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.642+0000 7f15efd83740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560664fd0 0x55f56066dc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.642+0000 7f15efd83740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560665d40 0x55f560669e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15efd83740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15efd83740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81aa490 con 0x55f56066e550 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15efd83740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81aca50 con 0x55f560664fd0 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15efd83740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81aaa90 con 0x55f560665d40 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.642 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50412/0 (socket says 172.21.15.73:50412) 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 -- 172.21.15.73:0/2463054264 learned_addr learned my addr 172.21.15.73:0/2463054264 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560664fd0 0x55f56066dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15dffff640 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560665d40 0x55f560669e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 -- 172.21.15.73:0/2463054264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560665d40 msgr2=0x55f560669e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560665d40 0x55f560669e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 -- 172.21.15.73:0/2463054264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560664fd0 msgr2=0x55f56066dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560664fd0 0x55f56066dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 -- 172.21.15.73:0/2463054264 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f5606642a0 con 0x55f56066e550 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15e759e640 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009f10 tx=0x7f15e001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fb817013efd81a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15dcff9640 1 -- 172.21.15.73:0/2463054264 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e001c760 con 0x55f56066e550 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15dcff9640 1 -- 172.21.15.73:0/2463054264 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15e001cde0 con 0x55f56066e550 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.643+0000 7f15dcff9640 1 -- 172.21.15.73:0/2463054264 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15e001f3f0 con 0x55f56066e550 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 -- 172.21.15.73:0/2463054264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 msgr2=0x55f56066a380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.643 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 crypto rx=0x7f15e0009f10 tx=0x7f15e001a000 comp rx=0 tx=0).stop 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 -- 172.21.15.73:0/2463054264 shutdown_connections 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f56066e550 0x55f56066a380 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560665d40 0x55f560669e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 --2- 172.21.15.73:0/2463054264 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f560664fd0 0x55f56066dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 -- 172.21.15.73:0/2463054264 >> 172.21.15.73:0/2463054264 conn(0x55f56062c5b0 msgr2=0x55f56065f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 -- 172.21.15.73:0/2463054264 shutdown_connections 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.644+0000 7f15efd83740 1 -- 172.21.15.73:0/2463054264 wait complete. 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 -- start start 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 Processor -- start 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560664fd0 0x55f560674230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f56066e550 0x55f560674cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81ab0d0 con 0x55f560665d40 2024-03-09T00:07:32.644 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81ac3f0 con 0x55f56066e550 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15e81ab730 con 0x55f560664fd0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15efd83740 10 client.0 _ll_register_callbacks cb 0x7f15d80093e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:50426/0 (socket says 172.21.15.73:50426) 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 -- 172.21.15.73:0/1495668199 learned_addr learned my addr 172.21.15.73:0/1495668199 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560664fd0 0x55f560674230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15e759e640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f56066e550 0x55f560674cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 -- 172.21.15.73:0/1495668199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560664fd0 msgr2=0x55f560674230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560664fd0 0x55f560674230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 -- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f56066e550 msgr2=0x55f560674cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f56066e550 0x55f560674cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.645+0000 7f15dffff640 1 -- 172.21.15.73:0/1495668199 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f5606d90c0 con 0x55f560665d40 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15dffff640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15d002c4e0 tx=0x7f15d002ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8703a8516632b218 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:32.645 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15d00d7070 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15d00bd550 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f15e001cde0 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15d00bf250 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 10 client.15783 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f5606da3b0 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f15e001cde0 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f15d8009d20 con 0x55f560665d40 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 10 client.15783 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 10 client.15783 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 20 client.15783 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f5606daa90 0x55f5606dceb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:32.646 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 4 client.15783 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:07:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.646+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f5605d0e80 con 0x55f5606daa90 2024-03-09T00:07:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.647+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f5606daa90 0x55f5606dceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.647+0000 7f15e6d9d640 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f5606daa90 0x55f5606dceb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=35cbf69a6c2712a4 server_cookie=e95d06d50f66ba14 in_seq=0 out_seq=0 2024-03-09T00:07:32.647 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.647+0000 7f15ce7fc640 10 client.15783 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15ce7fc640 1 -- 172.21.15.73:0/1495668199 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f5605d0e80 con 0x55f5606daa90 2024-03-09T00:07:32.649 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15ce7fc640 10 client.15783 handle_command_reply: tid=0 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 client.15783 shutdown 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 msgr2=0x55f560674770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f15d002c4e0 tx=0x7f15d002ce60 comp rx=0 tx=0).stop 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 shutdown_connections 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f5606daa90 0x55f5606dceb0 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f56066e550 0x55f560674cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f560665d40 0x55f560674770 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15ce7fc640 0 client.15783 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 --2- 172.21.15.73:0/1495668199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f560664fd0 0x55f560674230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 >> 172.21.15.73:0/1495668199 conn(0x55f56062c5b0 msgr2=0x55f560669d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 shutdown_connections 2024-03-09T00:07:32.650 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.654+0000 7f15efd83740 1 -- 172.21.15.73:0/1495668199 wait complete. 2024-03-09T00:07:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15efd83740 20 client.15783 trim_cache size 0 max 16384 2024-03-09T00:07:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:07:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:07:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:07:32.651 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 msgr2=0x7f15d40a4ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 0x7f15d40a4ff0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7f15e0005fd0 tx=0x7f15e0009ad0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 msgr2=0x7f15e819cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 crypto rx=0x7f15d0005940 tx=0x7f15d00043b0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 shutdown_connections 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15e8105190 0x7f15e8196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15d40a2b30 0x7f15d40a4ff0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15e8100e00 0x7f15e819cc00 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.652 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 --2- 172.21.15.73:0/1225272406 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f15e8100440 0x7f15e819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 >> 172.21.15.73:0/1225272406 conn(0x7f15e8077040 msgr2=0x7f15e80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 shutdown_connections 2024-03-09T00:07:32.653 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:32.655+0000 7f15ed1a9640 1 -- 172.21.15.73:0/1225272406 wait complete. 2024-03-09T00:07:34.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:34 smithi179 ceph-mon[36637]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-09T00:07:34.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:34 smithi073 ceph-mon[32309]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-09T00:07:34.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:34 smithi102 ceph-mon[36106]: pgmap v24: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 1.8 KiB/s wr, 3 op/s 2024-03-09T00:07:36.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:36 smithi179 ceph-mon[36637]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-09T00:07:36.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:36 smithi073 ceph-mon[32309]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-09T00:07:36.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:36 smithi102 ceph-mon[36106]: pgmap v25: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 1.3 KiB/s wr, 1 op/s 2024-03-09T00:07:38.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:38 smithi179 ceph-mon[36637]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-09T00:07:38.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:38 smithi073 ceph-mon[32309]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-09T00:07:38.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:38 smithi102 ceph-mon[36106]: pgmap v26: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 2.5 KiB/s wr, 4 op/s 2024-03-09T00:07:40.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:40 smithi179 ceph-mon[36637]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:40.932 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:40 smithi102 ceph-mon[36106]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:40.933 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:40 smithi073 ceph-mon[32309]: pgmap v27: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:42.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:42 smithi179 ceph-mon[36637]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:42.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:42 smithi073 ceph-mon[32309]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:42.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:42 smithi102 ceph-mon[36106]: pgmap v28: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:44.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:44 smithi073 ceph-mon[32309]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-09T00:07:44.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:44 smithi102 ceph-mon[36106]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-09T00:07:44.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:44 smithi179 ceph-mon[36637]: pgmap v29: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 1.2 KiB/s wr, 4 op/s 2024-03-09T00:07:45.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:07:45.720 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 -- 172.21.15.73:0/2742291151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860101520 msgr2=0x7fd860104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 --2- 172.21.15.73:0/2742291151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860101520 0x7fd860104e30 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 crypto rx=0x7fd850009900 tx=0x7fd85002bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 -- 172.21.15.73:0/2742291151 shutdown_connections 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 --2- 172.21.15.73:0/2742291151 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86010d120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 --2- 172.21.15.73:0/2742291151 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860101520 0x7fd860104e30 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 --2- 172.21.15.73:0/2742291151 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd860100b70 0x7fd860100f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 -- 172.21.15.73:0/2742291151 >> 172.21.15.73:0/2742291151 conn(0x7fd860076780 msgr2=0x7fd8600fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.753 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 -- 172.21.15.73:0/2742291151 shutdown_connections 2024-03-09T00:07:45.754 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.758+0000 7fd868553640 1 -- 172.21.15.73:0/2742291151 wait complete. 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 Processor -- start 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 -- start start 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86019a310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 0x7fd86019eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.756 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010f500 con 0x7fd860100b70 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010f380 con 0x7fd86019e640 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.759+0000 7fd868553640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd86010f680 con 0x7fd860105510 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd8662ef640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd8662ef640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46914/0 (socket says 172.21.15.73:46914) 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd8662ef640 1 -- 172.21.15.73:0/3946362372 learned_addr learned my addr 172.21.15.73:0/3946362372 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd866af0640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 0x7fd86019eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86019a310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 -- 172.21.15.73:0/3946362372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 msgr2=0x7fd86019eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 0x7fd86019eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 -- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 msgr2=0x7fd860199dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 -- 172.21.15.73:0/3946362372 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd86019f130 con 0x7fd860105510 2024-03-09T00:07:45.757 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd866af0640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 0x7fd86019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.760+0000 7fd865aee640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86019a310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd850036040 tx=0x7fd8500043b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.761+0000 7fd8577fe640 1 -- 172.21.15.73:0/3946362372 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850043020 con 0x7fd860105510 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.761+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd86019f3c0 con 0x7fd860105510 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.761+0000 7fd8577fe640 1 -- 172.21.15.73:0/3946362372 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8500054e0 con 0x7fd860105510 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.761+0000 7fd8577fe640 1 -- 172.21.15.73:0/3946362372 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd850003bd0 con 0x7fd860105510 2024-03-09T00:07:45.758 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.761+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8601a6be0 con 0x7fd860105510 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.763+0000 7fd86b214740 1 Processor -- start 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 -- start start 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900705450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900710940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5629006695f0 con 0x562900705d40 2024-03-09T00:07:45.761 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x562900612930 con 0x56290070e550 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd86b214740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x562900665690 con 0x562900704fd0 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd866af0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd866af0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46930/0 (socket says 172.21.15.73:46930) 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd866af0640 1 -- 172.21.15.73:0/969192179 learned_addr learned my addr 172.21.15.73:0/969192179 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd8577fe640 1 -- 172.21.15.73:0/3946362372 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fd850046020 con 0x7fd860105510 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.764+0000 7fd8662ef640 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900705450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.762 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 -- 172.21.15.73:0/969192179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 msgr2=0x562900705450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900705450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 -- 172.21.15.73:0/969192179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 msgr2=0x562900710940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900710940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 -- 172.21.15.73:0/969192179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56290037e210 con 0x562900705d40 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd866af0640 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd85800bfd0 tx=0x7fd85800b730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46958a12ece47b7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd854bfe640 1 -- 172.21.15.73:0/969192179 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858019020 con 0x562900705d40 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd854bfe640 1 -- 172.21.15.73:0/969192179 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd858015cf0 con 0x562900705d40 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd854bfe640 1 -- 172.21.15.73:0/969192179 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858014df0 con 0x562900705d40 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd8577fe640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 0x7fd84c0a50c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 0x7fd84c0a50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.765+0000 7fd8577fe640 1 -- 172.21.15.73:0/3946362372 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd8500f1490 con 0x7fd860105510 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 -- 172.21.15.73:0/969192179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 msgr2=0x562900704a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 0x7fd84c0a50c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00bfd0 tx=0x7fd85c00b4a0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.763 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 crypto rx=0x7fd85800bfd0 tx=0x7fd85800b730 comp rx=0 tx=0).stop 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 -- 172.21.15.73:0/969192179 shutdown_connections 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900710940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900705450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 --2- 172.21.15.73:0/969192179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x562900704a90 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 -- 172.21.15.73:0/969192179 >> 172.21.15.73:0/969192179 conn(0x5629006cc5b0 msgr2=0x5629006ffcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 -- 172.21.15.73:0/969192179 shutdown_connections 2024-03-09T00:07:45.764 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.766+0000 7fd86b214740 1 -- 172.21.15.73:0/969192179 wait complete. 2024-03-09T00:07:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 -- start start 2024-03-09T00:07:45.768 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 Processor -- start 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x56290070b660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 0x56290070bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd866af0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd866af0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46946/0 (socket says 172.21.15.73:46946) 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd866af0640 1 -- 172.21.15.73:0/844964595 learned_addr learned my addr 172.21.15.73:0/844964595 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x562900711280 con 0x56290070e550 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x562900625d90 con 0x562900705d40 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x562900711400 con 0x562900704fd0 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd8662ef640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x56290070b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.767+0000 7fd86b214740 10 client.0 _ll_register_callbacks cb 0x56290037da00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd865aee640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 0x56290070bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 -- 172.21.15.73:0/844964595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 msgr2=0x56290070b660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.769 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x56290070b660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 -- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 msgr2=0x56290070bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 0x56290070bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56290044a620 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd865aee640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 0x56290070bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd866af0640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd858000f80 tx=0x7fd85800beb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e119ca1343dfdb86 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd858019040 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd858004c40 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd858015cf0 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd858033050 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.768+0000 7fd86b214740 10 client.15795 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56290071d640 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd858015cf0 con 0x56290070e550 2024-03-09T00:07:45.770 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd858005640 con 0x56290070e550 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 10 client.15795 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 10 client.15795 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 20 client.15795 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56290071ea40 0x562900720e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 4 client.15795 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562900676b30 con 0x56290071ea40 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.769+0000 7fd8662ef640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56290071ea40 0x562900720e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.770+0000 7fd849ffb640 10 client.15795 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.770+0000 7fd8662ef640 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56290071ea40 0x562900720e30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d43a9dddd7bda65b server_cookie=16fbc07b78452f6d in_seq=0 out_seq=0 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd849ffb640 1 -- 172.21.15.73:0/844964595 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562900676b30 con 0x56290071ea40 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd849ffb640 10 client.15795 handle_command_reply: tid=0 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 client.15795 shutdown 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 msgr2=0x56290070c0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.771 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 crypto rx=0x7fd858000f80 tx=0x7fd85800beb0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 shutdown_connections 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56290071ea40 0x562900720e30 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56290070e550 0x56290070c0e0 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562900705d40 0x56290070bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 --2- 172.21.15.73:0/844964595 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x56290070b660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 >> 172.21.15.73:0/844964595 conn(0x5629006cc5b0 msgr2=0x562900701a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd849ffb640 0 client.15795 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 shutdown_connections 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 1 -- 172.21.15.73:0/844964595 wait complete. 2024-03-09T00:07:45.772 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.771+0000 7fd86b214740 20 client.15795 trim_cache size 0 max 16384 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.784+0000 7fd86b214740 1 Processor -- start 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.784+0000 7fd86b214740 1 -- start start 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900705d40 0x562900715670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900715bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.780 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601ae8a0 con 0x562900704fd0 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601b0e60 con 0x56290070e550 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd86b214740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601aeea0 con 0x562900705d40 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46948/0 (socket says 172.21.15.73:46948) 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 -- 172.21.15.73:0/3204727529 learned_addr learned my addr 172.21.15.73:0/3204727529 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd866af0640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900715bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd865aee640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900705d40 0x562900715670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 -- 172.21.15.73:0/3204727529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900705d40 msgr2=0x562900715670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900705d40 0x562900715670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 -- 172.21.15.73:0/3204727529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 msgr2=0x562900715bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900715bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.781 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd8662ef640 1 -- 172.21.15.73:0/3204727529 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629007064d0 con 0x562900704fd0 2024-03-09T00:07:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.785+0000 7fd866af0640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900715bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.786+0000 7fd8662ef640 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0114f0 tx=0x7fd85c019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae08d75b54c4d2a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.786+0000 7fd84bfff640 1 -- 172.21.15.73:0/3204727529 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c02e070 con 0x562900704fd0 2024-03-09T00:07:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.786+0000 7fd84bfff640 1 -- 172.21.15.73:0/3204727529 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd85c029890 con 0x562900704fd0 2024-03-09T00:07:45.782 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.786+0000 7fd84bfff640 1 -- 172.21.15.73:0/3204727529 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd85c003d60 con 0x562900704fd0 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- 172.21.15.73:0/3204727529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 msgr2=0x56290070dd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 crypto rx=0x7fd85c0114f0 tx=0x7fd85c019000 comp rx=0 tx=0).stop 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- 172.21.15.73:0/3204727529 shutdown_connections 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900715bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900705d40 0x562900715670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 --2- 172.21.15.73:0/3204727529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900704fd0 0x56290070dd90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- 172.21.15.73:0/3204727529 >> 172.21.15.73:0/3204727529 conn(0x5629006cc5b0 msgr2=0x5629006ff7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.785 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- 172.21.15.73:0/3204727529 shutdown_connections 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- 172.21.15.73:0/3204727529 wait complete. 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 -- start start 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.787+0000 7fd86b214740 1 Processor -- start 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900711fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900712a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601af4e0 con 0x562900705d40 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601b0800 con 0x56290070e550 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd86b214740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8601afb40 con 0x562900704fd0 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46964/0 (socket says 172.21.15.73:46964) 2024-03-09T00:07:45.786 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 -- 172.21.15.73:0/16433704 learned_addr learned my addr 172.21.15.73:0/16433704 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd8662ef640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900711fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd866af0640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900712a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 -- 172.21.15.73:0/16433704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 msgr2=0x562900711fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900711fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 -- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 msgr2=0x562900712a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900712a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.788+0000 7fd865aee640 1 -- 172.21.15.73:0/16433704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562900774b40 con 0x562900705d40 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 10 client.0 _ll_register_callbacks cb 0x7fd85800bdc0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd865aee640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd8500eee50 tx=0x7fd8500b9830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23f22a2a4f94564f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd849ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd85c029890 con 0x562900705d40 2024-03-09T00:07:45.787 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd849ffb640 1 -- 172.21.15.73:0/16433704 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8500d6060 con 0x562900705d40 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd849ffb640 1 -- 172.21.15.73:0/16433704 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd8500c5930 con 0x562900705d40 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd849ffb640 1 -- 172.21.15.73:0/16433704 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd8500bcf80 con 0x562900705d40 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 10 client.15801 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({fsmap=20}) v3 -- 0x562900775ca0 con 0x562900705d40 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd849ffb640 1 -- 172.21.15.73:0/16433704 <== mon.0 v2:172.21.15.73:3300/0 4 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd8500ee9b0 con 0x562900705d40 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 10 client.15801 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 10 client.15801 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 20 client.15801 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562900776280 0x5629007786a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 4 client.15801 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:07:45.788 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.789+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x562900670e80 con 0x562900776280 2024-03-09T00:07:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.790+0000 7fd8662ef640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562900776280 0x5629007786a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.790+0000 7fd849ffb640 10 client.15801 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.789 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.790+0000 7fd8662ef640 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562900776280 0x5629007786a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e2a7d6936754fa1e server_cookie=4abb446cee26e1c3 in_seq=0 out_seq=0 2024-03-09T00:07:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd849ffb640 1 -- 172.21.15.73:0/16433704 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x562900670e80 con 0x562900776280 2024-03-09T00:07:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd849ffb640 10 client.15801 handle_command_reply: tid=0 2024-03-09T00:07:45.790 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 client.15801 shutdown 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 msgr2=0x5629007124e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 crypto rx=0x7fd8500eee50 tx=0x7fd8500b9830 comp rx=0 tx=0).stop 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 shutdown_connections 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562900776280 0x5629007786a0 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56290070e550 0x562900712a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562900705d40 0x5629007124e0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 --2- 172.21.15.73:0/16433704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562900704fd0 0x562900711fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.791 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd849ffb640 0 client.15801 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 >> 172.21.15.73:0/16433704 conn(0x5629006cc5b0 msgr2=0x562900709ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 shutdown_connections 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.794+0000 7fd86b214740 1 -- 172.21.15.73:0/16433704 wait complete. 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd86b214740 20 client.15801 trim_cache size 0 max 16384 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:07:45.792 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:07:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 msgr2=0x7fd84c0a50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 0x7fd84c0a50c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fd85c00bfd0 tx=0x7fd85c00b4a0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.793 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 msgr2=0x7fd86019a310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86019a310 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd850036040 tx=0x7fd8500043b0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 shutdown_connections 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd86019e640 0x7fd86019eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd860105510 0x7fd86019a310 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd84c0a2c00 0x7fd84c0a50c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 --2- 172.21.15.73:0/3946362372 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd860100b70 0x7fd860199dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 >> 172.21.15.73:0/3946362372 conn(0x7fd860076780 msgr2=0x7fd860103bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.795+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 shutdown_connections 2024-03-09T00:07:45.794 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:45.796+0000 7fd868553640 1 -- 172.21.15.73:0/3946362372 wait complete. 2024-03-09T00:07:45.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 -- 172.21.15.73:0/2930502131 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04100d80 msgr2=0x7fbd04104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 --2- 172.21.15.73:0/2930502131 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04100d80 0x7fbd04104dc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 crypto rx=0x7fbcf8009900 tx=0x7fbcf802bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:45.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 -- 172.21.15.73:0/2930502131 shutdown_connections 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 --2- 172.21.15.73:0/2930502131 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041054a0 0x7fbd0410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 --2- 172.21.15.73:0/2930502131 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04100d80 0x7fbd04104dc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 --2- 172.21.15.73:0/2930502131 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041003d0 0x7fbd041007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 -- 172.21.15.73:0/2930502131 >> 172.21.15.73:0/2930502131 conn(0x7fbd040772f0 msgr2=0x7fbd04077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 -- 172.21.15.73:0/2930502131 shutdown_connections 2024-03-09T00:07:45.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.868+0000 7fbd0a962640 1 -- 172.21.15.73:0/2930502131 wait complete. 2024-03-09T00:07:45.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.869+0000 7fbd0a962640 1 Processor -- start 2024-03-09T00:07:45.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.869+0000 7fbd0a962640 1 -- start start 2024-03-09T00:07:45.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.869+0000 7fbd0a962640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 0x7fbd0419c690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.869+0000 7fbd0a962640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 0x7fbd0419cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.869+0000 7fbd0a962640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd0a962640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd04110560 con 0x7fbd04196800 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd0a962640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041103e0 con 0x7fbd041003d0 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd0a962640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041106e0 con 0x7fbd041054a0 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46990/0 (socket says 172.21.15.73:46990) 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd08eff640 1 -- 172.21.15.73:0/711747482 learned_addr learned my addr 172.21.15.73:0/711747482 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 0x7fbd0419c690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd037fe640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 0x7fbd0419cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 -- 172.21.15.73:0/711747482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 msgr2=0x7fbd0419cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 0x7fbd0419cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 -- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 msgr2=0x7fbd04196cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 -- 172.21.15.73:0/711747482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd04197390 con 0x7fbd041003d0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd08eff640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd037fe640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 0x7fbd0419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.870+0000 7fbd03fff640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 0x7fbd0419c690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009f90 tx=0x7fbcf400edd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.871+0000 7fbd017fa640 1 -- 172.21.15.73:0/711747482 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf40159e0 con 0x7fbd041003d0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.871+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd04197680 con 0x7fbd041003d0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.871+0000 7fbd017fa640 1 -- 172.21.15.73:0/711747482 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcf400cd90 con 0x7fbd041003d0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.871+0000 7fbd017fa640 1 -- 172.21.15.73:0/711747482 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf40055f0 con 0x7fbd041003d0 2024-03-09T00:07:45.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.871+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd04068750 con 0x7fbd041003d0 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.873+0000 7fbd017fa640 1 -- 172.21.15.73:0/711747482 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fbcf400c8b0 con 0x7fbd041003d0 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.873+0000 7fbd017fa640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 0x7fbcec0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd017fa640 1 -- 172.21.15.73:0/711747482 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbcf40d3790 con 0x7fbd041003d0 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd037fe640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 0x7fbcec0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 Processor -- start 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 -- start start 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0908bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd037fe640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 0x7fbcec0a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcf802c240 tx=0x7fbcf802df90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09094720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee08feee00 con 0x55ee09094c60 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee08f97b10 con 0x55ee0908c080 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.874+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee08feaea0 con 0x55ee0908b760 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46994/0 (socket says 172.21.15.73:46994) 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 -- 172.21.15.73:0/204607320 learned_addr learned my addr 172.21.15.73:0/204607320 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd03fff640 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0908bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd037fe640 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09094720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 -- 172.21.15.73:0/204607320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 msgr2=0x55ee0908bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0908bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 -- 172.21.15.73:0/204607320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 msgr2=0x55ee09094720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09094720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 -- 172.21.15.73:0/204607320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee08dcd7d0 con 0x55ee09094c60 2024-03-09T00:07:45.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbd08eff640 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00f790 tx=0x7fbcfc00fb90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82390e0056aa5d1e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbcea7fc640 1 -- 172.21.15.73:0/204607320 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc004d20 con 0x55ee09094c60 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbcea7fc640 1 -- 172.21.15.73:0/204607320 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc00de00 con 0x55ee09094c60 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.875+0000 7fbcea7fc640 1 -- 172.21.15.73:0/204607320 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc010430 con 0x55ee09094c60 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 -- 172.21.15.73:0/204607320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 msgr2=0x55ee09097050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00f790 tx=0x7fbcfc00fb90 comp rx=0 tx=0).stop 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 -- 172.21.15.73:0/204607320 shutdown_connections 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09097050 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09094720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/204607320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0908bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 -- 172.21.15.73:0/204607320 >> 172.21.15.73:0/204607320 conn(0x55ee09085e90 msgr2=0x55ee09086270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 -- 172.21.15.73:0/204607320 shutdown_connections 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.876+0000 7fbd0d61e740 1 -- 172.21.15.73:0/204607320 wait complete. 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.877+0000 7fbd0d61e740 1 -- start start 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.877+0000 7fbd0d61e740 1 Processor -- start 2024-03-09T00:07:45.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.877+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908b760 0x55ee09094380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee09094c60 0x55ee09091c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47006/0 (socket says 172.21.15.73:47006) 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 -- 172.21.15.73:0/1327107179 learned_addr learned my addr 172.21.15.73:0/1327107179 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee09097a50 con 0x55ee0908c080 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee08fab0a0 con 0x55ee0908b760 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee09097bd0 con 0x55ee09094c60 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd03fff640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908b760 0x55ee09094380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd0d61e740 10 client.0 _ll_register_callbacks cb 0x55ee08faaa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 -- 172.21.15.73:0/1327107179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee09094c60 msgr2=0x55ee09091c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee09094c60 0x55ee09091c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 -- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908b760 msgr2=0x55ee09094380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908b760 0x55ee09094380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee08d033f0 con 0x55ee0908c080 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbd037fe640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbcf802c820 tx=0x7fbcf8009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=669c4c2324259c9d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbce37fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf8051030 con 0x55ee0908c080 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.878+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbcf803a040 con 0x55ee0908c080 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcfc00de00 con 0x55ee0908c080 2024-03-09T00:07:45.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcf8004ee0 con 0x55ee0908c080 2024-03-09T00:07:45.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbd0d61e740 10 client.15813 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:45.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ee08cc30f0 con 0x55ee0908c080 2024-03-09T00:07:45.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc00de00 con 0x55ee0908c080 2024-03-09T00:07:45.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.879+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbcf8003b10 con 0x55ee0908c080 2024-03-09T00:07:45.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 10 client.15813 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:45.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 10 client.15813 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:45.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 20 client.15813 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:45.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090a6b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 4 client.15813 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee08ffc340 con 0x55ee090a46b0 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.880+0000 7fbd03fff640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090a6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbd03fff640 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090a6b70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a37413722583a9ce server_cookie=e1a6558cd7df59b5 in_seq=0 out_seq=0 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbce37fe640 10 client.15813 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbce37fe640 1 -- 172.21.15.73:0/1327107179 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ee08ffc340 con 0x55ee090a46b0 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbce37fe640 10 client.15813 handle_command_reply: tid=0 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbd0d61e740 1 client.15813 shutdown 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 msgr2=0x55ee09091710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.881+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 crypto rx=0x7fbcf802c820 tx=0x7fbcf8009260 comp rx=0 tx=0).stop 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 shutdown_connections 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090a6b70 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbce37fe640 0 client.15813 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee09094c60 0x55ee09091c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908c080 0x55ee09091710 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/1327107179 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908b760 0x55ee09094380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 >> 172.21.15.73:0/1327107179 conn(0x55ee09085e90 msgr2=0x55ee09087be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 shutdown_connections 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 1 -- 172.21.15.73:0/1327107179 wait complete. 2024-03-09T00:07:45.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.882+0000 7fbd0d61e740 20 client.15813 trim_cache size 0 max 16384 2024-03-09T00:07:45.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.911+0000 7fbd0d61e740 1 Processor -- start 2024-03-09T00:07:45.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.911+0000 7fbd0d61e740 1 -- start start 2024-03-09T00:07:45.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.911+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.911+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 0x55ee09093e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.911+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee09094c60 0x55ee09094380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041aa450 con 0x55ee0908b760 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041aca10 con 0x55ee09094c60 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd0d61e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041aaa50 con 0x55ee0908c080 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47010/0 (socket says 172.21.15.73:47010) 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 -- 172.21.15.73:0/42055059 learned_addr learned my addr 172.21.15.73:0/42055059 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd037fe640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 0x55ee09093e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd08eff640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee09094c60 0x55ee09094380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 -- 172.21.15.73:0/42055059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 msgr2=0x55ee09093e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 0x55ee09093e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 -- 172.21.15.73:0/42055059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee09094c60 msgr2=0x55ee09094380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee09094c60 0x55ee09094380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 -- 172.21.15.73:0/42055059 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee0908aae0 con 0x55ee0908b760 2024-03-09T00:07:45.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd037fe640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 0x55ee09093e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:45.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbd03fff640 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbcf409f5c0 tx=0x7fbcf408a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8bcfd53fc4357dcf server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbce9ffb640 1 -- 172.21.15.73:0/42055059 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf4023040 con 0x55ee0908b760 2024-03-09T00:07:45.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbce9ffb640 1 -- 172.21.15.73:0/42055059 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcf400cba0 con 0x55ee0908b760 2024-03-09T00:07:45.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.912+0000 7fbce9ffb640 1 -- 172.21.15.73:0/42055059 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcf409c020 con 0x55ee0908b760 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 -- 172.21.15.73:0/42055059 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 msgr2=0x55ee0908ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 crypto rx=0x7fbcf409f5c0 tx=0x7fbcf408a040 comp rx=0 tx=0).stop 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 -- 172.21.15.73:0/42055059 shutdown_connections 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee09094c60 0x55ee09094380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908c080 0x55ee09093e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/42055059 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee0908b760 0x55ee0908ccb0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 -- 172.21.15.73:0/42055059 >> 172.21.15.73:0/42055059 conn(0x55ee09085e90 msgr2=0x55ee0908c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 -- 172.21.15.73:0/42055059 shutdown_connections 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.913+0000 7fbd0d61e740 1 -- 172.21.15.73:0/42055059 wait complete. 2024-03-09T00:07:45.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.914+0000 7fbd0d61e740 1 -- start start 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.914+0000 7fbd0d61e740 1 Processor -- start 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0909c120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09098520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47022/0 (socket says 172.21.15.73:47022) 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 -- 172.21.15.73:0/3300899928 learned_addr learned my addr 172.21.15.73:0/3300899928 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd037fe640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09098520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ab090 con 0x55ee09094c60 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ac3b0 con 0x55ee0908c080 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd041ab6f0 con 0x55ee0908b760 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd03fff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0909c120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd0d61e740 10 client.0 _ll_register_callbacks cb 0x7fbcf8004b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:45.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 -- 172.21.15.73:0/3300899928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 msgr2=0x55ee0909c120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0909c120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 -- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 msgr2=0x55ee09098520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09098520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd08eff640 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee09099260 con 0x55ee09094c60 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.915+0000 7fbd037fe640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09098520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbd08eff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00f790 tx=0x7fbcfc00db50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21498d3a42882635 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbce37fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcfc00de50 con 0x55ee09094c60 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbcfc005040 con 0x55ee09094c60 2024-03-09T00:07:45.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbcf400cba0 con 0x55ee09094c60 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbcfc030050 con 0x55ee09094c60 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbd0d61e740 10 client.15819 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ee090fbbb0 con 0x55ee09094c60 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.916+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcfc027070 con 0x55ee09094c60 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbcfc022630 con 0x55ee09094c60 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 10 client.15819 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 10 client.15819 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 20 client.15819 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090fe0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 4 client.15819 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7fbcf400cba0 con 0x55ee090a46b0 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbd03fff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090fe0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:45.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.917+0000 7fbce37fe640 10 client.15819 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd03fff640 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090fe0b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2c8dc37fec05b400 server_cookie=9d5b2d70249e20f9 in_seq=0 out_seq=0 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbce37fe640 1 -- 172.21.15.73:0/3300899928 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x7fbcf400cba0 con 0x55ee090a46b0 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbce37fe640 10 client.15819 handle_command_reply: tid=0 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 client.15819 shutdown 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 msgr2=0x55ee09098a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 crypto rx=0x7fbcfc00f790 tx=0x7fbcfc00db50 comp rx=0 tx=0).stop 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 shutdown_connections 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee090a46b0 0x55ee090fe0b0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbce37fe640 0 client.15819 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee09094c60 0x55ee09098a60 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee0908c080 0x55ee09098520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 --2- 172.21.15.73:0/3300899928 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee0908b760 0x55ee0909c120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 >> 172.21.15.73:0/3300899928 conn(0x55ee09085e90 msgr2=0x55ee0908ffe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.918+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 shutdown_connections 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0d61e740 1 -- 172.21.15.73:0/3300899928 wait complete. 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0d61e740 20 client.15819 trim_cache size 0 max 16384 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:07:45.919 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:45.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 msgr2=0x7fbcec0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 0x7fbcec0a4f50 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fbcf802c240 tx=0x7fbcf802df90 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 msgr2=0x7fbd0419c690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 0x7fbd0419c690 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009f90 tx=0x7fbcf400edd0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 shutdown_connections 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbcec0a2a90 0x7fbcec0a4f50 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd04196800 0x7fbd04196cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd041054a0 0x7fbd0419cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 --2- 172.21.15.73:0/711747482 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd041003d0 0x7fbd0419c690 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:45.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.919+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 >> 172.21.15.73:0/711747482 conn(0x7fbd040772f0 msgr2=0x7fbd040fe1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:45.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.920+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 shutdown_connections 2024-03-09T00:07:45.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:45.920+0000 7fbd0a962640 1 -- 172.21.15.73:0/711747482 wait complete. 2024-03-09T00:07:45.926 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:07:45.926 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 -- 172.21.15.73:0/3731069334 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f414810c8c0 msgr2=0x7f4148110900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 --2- 172.21.15.73:0/3731069334 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f414810c8c0 0x7f4148110900 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 crypto rx=0x7f4138009900 tx=0x7f413802bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 -- 172.21.15.73:0/3731069334 shutdown_connections 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 --2- 172.21.15.73:0/3731069334 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4148110fe0 0x7f4148118bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 --2- 172.21.15.73:0/3731069334 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f414810c8c0 0x7f4148110900 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 --2- 172.21.15.73:0/3731069334 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f414810bf10 0x7f414810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 -- 172.21.15.73:0/3731069334 >> 172.21.15.73:0/3731069334 conn(0x7f414806c350 msgr2=0x7f414810a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 -- 172.21.15.73:0/3731069334 shutdown_connections 2024-03-09T00:07:46.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.077+0000 7f414fc62640 1 -- 172.21.15.73:0/3731069334 wait complete. 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 Processor -- start 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 -- start start 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4148110fe0 0x7f41481a2b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414811c0a0 con 0x7f41481a6e70 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414811bf20 con 0x7f4148110fe0 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414fc62640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f414811c220 con 0x7f414810bf10 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414d9fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414e1ff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414e1ff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47036/0 (socket says 172.21.15.73:47036) 2024-03-09T00:07:46.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414e1ff640 1 -- 172.21.15.73:0/1536408828 learned_addr learned my addr 172.21.15.73:0/1536408828 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.078+0000 7f414d9fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:34974/0 (socket says 172.21.15.73:34974) 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d1fd640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4148110fe0 0x7f41481a2b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 -- 172.21.15.73:0/1536408828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4148110fe0 msgr2=0x7f41481a2b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4148110fe0 0x7f41481a2b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 -- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 msgr2=0x7f41481a7320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 -- 172.21.15.73:0/1536408828 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41481a7960 con 0x7f414810bf10 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414e1ff640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414d9fe640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4144005fd0 tx=0x7f414400dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144009c30 con 0x7f414810bf10 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41481a7c50 con 0x7f414810bf10 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.079+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4144009dd0 con 0x7f414810bf10 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.080+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4144010460 con 0x7f414810bf10 2024-03-09T00:07:46.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.080+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41481af460 con 0x7f414810bf10 2024-03-09T00:07:46.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.081+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f414810d3e0 con 0x7f414810bf10 2024-03-09T00:07:46.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.082+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f41440035f0 con 0x7f414810bf10 2024-03-09T00:07:46.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.083+0000 7f413effd640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 0x7f41340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.083+0000 7f414d1fd640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 0x7f41340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.083+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41440cf000 con 0x7f414810bf10 2024-03-09T00:07:46.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.083+0000 7f414d1fd640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 0x7f41340a4e30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f413802c240 tx=0x7f413802df90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.088+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f4144017020 con 0x7f414810bf10 2024-03-09T00:07:46.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:46 smithi179 ceph-mon[36637]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:46.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:46 smithi073 ceph-mon[32309]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:46.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.434+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f41481a3900 con 0x7f414810bf10 2024-03-09T00:07:46.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.435+0000 7f413effd640 1 -- 172.21.15.73:0/1536408828 <== mon.1 v2:172.21.15.179:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f414409b130 con 0x7f414810bf10 2024-03-09T00:07:46.430 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:07:46.430 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:07:46.430 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:07:46.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.437+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 msgr2=0x7f41340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.437+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 0x7f41340a4e30 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f413802c240 tx=0x7f413802df90 comp rx=0 tx=0).stop 2024-03-09T00:07:46.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.437+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 msgr2=0x7f41481a2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.437+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f4144005fd0 tx=0x7f414400dde0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 shutdown_connections 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f41340a2970 0x7f41340a4e30 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41481a6e70 0x7f41481a7320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4148110fe0 0x7f41481a2b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 --2- 172.21.15.73:0/1536408828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f414810bf10 0x7f41481a2600 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 >> 172.21.15.73:0/1536408828 conn(0x7f414806c350 msgr2=0x7f414810f2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 shutdown_connections 2024-03-09T00:07:46.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.438+0000 7f414fc62640 1 -- 172.21.15.73:0/1536408828 wait complete. 2024-03-09T00:07:46.448 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:07:46.448 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:07:46.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:46 smithi102 ceph-mon[36106]: pgmap v30: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 1.2 KiB/s wr, 3 op/s 2024-03-09T00:07:46.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 -- 172.21.15.73:0/626740252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1054b0 msgr2=0x7faf2c10d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 --2- 172.21.15.73:0/626740252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1054b0 0x7faf2c10d080 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 crypto rx=0x7faf24009ca0 tx=0x7faf2402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 -- 172.21.15.73:0/626740252 shutdown_connections 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 --2- 172.21.15.73:0/626740252 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1054b0 0x7faf2c10d080 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 --2- 172.21.15.73:0/626740252 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c100d90 0x7faf2c104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 --2- 172.21.15.73:0/626740252 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c1003e0 0x7faf2c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 -- 172.21.15.73:0/626740252 >> 172.21.15.73:0/626740252 conn(0x7faf2c0773a0 msgr2=0x7faf2c0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.615+0000 7faf3157d640 1 -- 172.21.15.73:0/626740252 shutdown_connections 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.616+0000 7faf3157d640 1 -- 172.21.15.73:0/626740252 wait complete. 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.616+0000 7faf3157d640 1 Processor -- start 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.616+0000 7faf3157d640 1 -- start start 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c1054b0 0x7faf2c196c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c110570 con 0x7faf2c1003e0 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1103f0 con 0x7faf2c197180 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf3157d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1106f0 con 0x7faf2c1054b0 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:45250/0 (socket says 172.21.15.73:45250) 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47044/0 (socket says 172.21.15.73:47044) 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2b7fe640 1 -- 172.21.15.73:0/1098275506 learned_addr learned my addr 172.21.15.73:0/1098275506 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf23fff640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c1054b0 0x7faf2c196c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 -- 172.21.15.73:0/1098275506 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c1054b0 msgr2=0x7faf2c196c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c1054b0 0x7faf2c196c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 -- 172.21.15.73:0/1098275506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 msgr2=0x7faf2c1a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 -- 172.21.15.73:0/1098275506 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf2c1a6340 con 0x7faf2c1003e0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.617+0000 7faf2affd640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00d950 tx=0x7faf1c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.618+0000 7faf28ff9640 1 -- 172.21.15.73:0/1098275506 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c009bd0 con 0x7faf2c1003e0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.618+0000 7faf28ff9640 1 -- 172.21.15.73:0/1098275506 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf1c009d70 con 0x7faf2c1003e0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.618+0000 7faf28ff9640 1 -- 172.21.15.73:0/1098275506 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c010460 con 0x7faf2c1003e0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.618+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf2c198970 con 0x7faf2c1003e0 2024-03-09T00:07:46.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.618+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf2c198e70 con 0x7faf2c1003e0 2024-03-09T00:07:46.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.621+0000 7faf28ff9640 1 -- 172.21.15.73:0/1098275506 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7faf1c003e80 con 0x7faf2c1003e0 2024-03-09T00:07:46.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.622+0000 7faf34231740 1 Processor -- start 2024-03-09T00:07:46.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.622+0000 7faf34231740 1 -- start start 2024-03-09T00:07:46.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.622+0000 7faf28ff9640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 0x7faf180a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c2ac0 0x5624ac5cb160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5cb6a0 0x5624ac5cda90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac589360 con 0x5624ac5cb6a0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac525ec0 con 0x5624ac5c21a0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf34231740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac4ced30 con 0x5624ac5c2ac0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 0x7faf180a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf28ff9640 1 -- 172.21.15.73:0/1098275506 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faf1c0cee20 con 0x7faf2c1003e0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:45260/0 (socket says 172.21.15.73:45260) 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 -- 172.21.15.73:0/1617948199 learned_addr learned my addr 172.21.15.73:0/1617948199 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf23fff640 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c2ac0 0x5624ac5cb160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 -- 172.21.15.73:0/1617948199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c2ac0 msgr2=0x5624ac5cb160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c2ac0 0x5624ac5cb160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 -- 172.21.15.73:0/1617948199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5cb6a0 msgr2=0x5624ac5cda90 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5cb6a0 0x5624ac5cda90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 -- 172.21.15.73:0/1617948199 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ac4cdbc0 con 0x5624ac5c21a0 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 0x7faf180a4fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf24009a60 tx=0x7faf24042040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.623+0000 7faf2affd640 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0973b0 tx=0x7faf1c0035f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9c6f9f6756d8766d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.624+0000 7faf217fa640 1 -- 172.21.15.73:0/1617948199 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c020040 con 0x5624ac5c21a0 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.624+0000 7faf217fa640 1 -- 172.21.15.73:0/1617948199 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf1c0a3640 con 0x5624ac5c21a0 2024-03-09T00:07:46.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.624+0000 7faf217fa640 1 -- 172.21.15.73:0/1617948199 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf1c0cc5a0 con 0x5624ac5c21a0 2024-03-09T00:07:46.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- 172.21.15.73:0/1617948199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 msgr2=0x5624ac5c2580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7faf1c0973b0 tx=0x7faf1c0035f0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- 172.21.15.73:0/1617948199 shutdown_connections 2024-03-09T00:07:46.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5cb6a0 0x5624ac5cda90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c2ac0 0x5624ac5cb160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 --2- 172.21.15.73:0/1617948199 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5c21a0 0x5624ac5c2580 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- 172.21.15.73:0/1617948199 >> 172.21.15.73:0/1617948199 conn(0x5624ac5bc8d0 msgr2=0x5624ac5bccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- 172.21.15.73:0/1617948199 shutdown_connections 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- 172.21.15.73:0/1617948199 wait complete. 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 -- start start 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.625+0000 7faf34231740 1 Processor -- start 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5c8760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c8ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac4c44a0 con 0x5624ac5c2ac0 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac521f60 con 0x5624ac5cb6a0 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5624ac5ce490 con 0x5624ac5c21a0 2024-03-09T00:07:46.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf23fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c8ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:45270/0 (socket says 172.21.15.73:45270) 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 -- 172.21.15.73:0/3102391065 learned_addr learned my addr 172.21.15.73:0/3102391065 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2affd640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5c8760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf34231740 10 client.0 _ll_register_callbacks cb 0x5624ac51fd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 -- 172.21.15.73:0/3102391065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 msgr2=0x5624ac5c8760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5c8760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 -- 172.21.15.73:0/3102391065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 msgr2=0x5624ac5c8ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c8ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf2b7fe640 1 -- 172.21.15.73:0/3102391065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624ac23a3f0 con 0x5624ac5cb6a0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.626+0000 7faf23fff640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf24037040 tx=0x7faf240095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3b89f5c7cea7242f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf127fc640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24040d90 con 0x5624ac5cb6a0 2024-03-09T00:07:46.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf1c0a3640 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faf24003970 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf24032a40 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf34231740 10 client.25271 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.627+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5624ac3047d0 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf1c0a3640 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faf2404a070 con 0x5624ac5cb6a0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 10 client.25271 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 10 client.25271 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 20 client.25271 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac5ddd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 4 client.25271 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5624ac533400 con 0x5624ac5db930 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.628+0000 7faf2affd640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac5ddd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf127fc640 10 client.25271 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf2affd640 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac5ddd20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a44da2d538dbcd8 server_cookie=99a5a63f2df6f601 in_seq=0 out_seq=0 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf127fc640 1 -- 172.21.15.73:0/3102391065 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5624ac533400 con 0x5624ac5db930 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf127fc640 10 client.25271 handle_command_reply: tid=0 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf34231740 1 client.25271 shutdown 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 msgr2=0x5624ac5c91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7faf24037040 tx=0x7faf240095c0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 shutdown_connections 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.629+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac5ddd20 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c91e0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf127fc640 0 client.25271 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c8ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 --2- 172.21.15.73:0/3102391065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5c8760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 >> 172.21.15.73:0/3102391065 conn(0x5624ac5bc8d0 msgr2=0x5624ac5be620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 shutdown_connections 2024-03-09T00:07:46.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 1 -- 172.21.15.73:0/3102391065 wait complete. 2024-03-09T00:07:46.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.630+0000 7faf34231740 20 client.25271 trim_cache size 0 max 16384 2024-03-09T00:07:46.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.653+0000 7faf34231740 1 Processor -- start 2024-03-09T00:07:46.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.653+0000 7faf34231740 1 -- start start 2024-03-09T00:07:46.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.653+0000 7faf34231740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf34231740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf34231740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c5f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf34231740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1aa540 con 0x5624ac5c2ac0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf34231740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1acae0 con 0x5624ac5cb6a0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf23fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf34231740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1aab40 con 0x5624ac5c21a0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c5f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf23fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47060/0 (socket says 172.21.15.73:47060) 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:35026/0 (socket says 172.21.15.73:35026) 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf23fff640 1 -- 172.21.15.73:0/4198502375 learned_addr learned my addr 172.21.15.73:0/4198502375 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 -- 172.21.15.73:0/4198502375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 msgr2=0x5624ac5cae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 -- 172.21.15.73:0/4198502375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 msgr2=0x5624ac5c59e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2b7fe640 1 -- 172.21.15.73:0/4198502375 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5624ac5bc400 con 0x5624ac5cb6a0 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf2affd640 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.654+0000 7faf23fff640 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.655+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c5f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf24032480 tx=0x7faf2402c7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cad48d00e59ba5d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.655+0000 7faf20ff9640 1 -- 172.21.15.73:0/4198502375 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24040b90 con 0x5624ac5cb6a0 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.655+0000 7faf20ff9640 1 -- 172.21.15.73:0/4198502375 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf240047b0 con 0x5624ac5cb6a0 2024-03-09T00:07:46.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.655+0000 7faf20ff9640 1 -- 172.21.15.73:0/4198502375 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf24034a50 con 0x5624ac5cb6a0 2024-03-09T00:07:46.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 -- 172.21.15.73:0/4198502375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 msgr2=0x5624ac5c5f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c5f20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faf24032480 tx=0x7faf2402c7f0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 -- 172.21.15.73:0/4198502375 shutdown_connections 2024-03-09T00:07:46.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5c5f20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5c59e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 --2- 172.21.15.73:0/4198502375 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5cae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 -- 172.21.15.73:0/4198502375 >> 172.21.15.73:0/4198502375 conn(0x5624ac5bc8d0 msgr2=0x5624ac5e2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 -- 172.21.15.73:0/4198502375 shutdown_connections 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.656+0000 7faf34231740 1 -- 172.21.15.73:0/4198502375 wait complete. 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 -- start start 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 Processor -- start 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5d1110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf2affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5d1110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf23fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf2affd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5d1110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:35030/0 (socket says 172.21.15.73:35030) 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf23fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47070/0 (socket says 172.21.15.73:47070) 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf2affd640 1 -- 172.21.15.73:0/1570290192 learned_addr learned my addr 172.21.15.73:0/1570290192 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:46.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5d1b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1ab160 con 0x5624ac5c2ac0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1ac480 con 0x5624ac5cb6a0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.657+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf2c1ab7c0 con 0x5624ac5c21a0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf34231740 10 client.0 _ll_register_callbacks cb 0x7faf1c00df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5d1b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 -- 172.21.15.73:0/1570290192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 msgr2=0x5624ac5d1110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5d1110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 -- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 msgr2=0x5624ac5d1b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5d1b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5624ac631e40 con 0x5624ac5c2ac0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf2b7fe640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf23fff640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5624ac5bdcd0 tx=0x7faf14003d00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c058e1c91545a31 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf127fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:46.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.658+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faf14021060 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faf140118f0 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faf240047b0 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7faf14028050 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 10 client.15840 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5624ac633130 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faf240047b0 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faf1401b070 con 0x5624ac5c2ac0 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 10 client.15840 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 10 client.15840 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.659+0000 7faf34231740 20 client.15840 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac635880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf34231740 4 client.15840 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5624ac52d750 con 0x5624ac5db930 2024-03-09T00:07:46.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf2affd640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac635880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf2affd640 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac635880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=828bd5134a76a1f1 server_cookie=f71ca9c5992cf333 in_seq=0 out_seq=0 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.660+0000 7faf127fc640 10 client.15840 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf127fc640 1 -- 172.21.15.73:0/1570290192 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5624ac52d750 con 0x5624ac5db930 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf127fc640 10 client.15840 handle_command_reply: tid=0 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 client.15840 shutdown 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 msgr2=0x5624ac5d1650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 crypto rx=0x5624ac5bdcd0 tx=0x7faf14003d00 comp rx=0 tx=0).stop 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 shutdown_connections 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5624ac5db930 0x5624ac635880 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5624ac5cb6a0 0x5624ac5d1b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5624ac5c2ac0 0x5624ac5d1650 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 --2- 172.21.15.73:0/1570290192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5624ac5c21a0 0x5624ac5d1110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf127fc640 0 client.15840 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 >> 172.21.15.73:0/1570290192 conn(0x5624ac5bc8d0 msgr2=0x5624ac5bb100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 shutdown_connections 2024-03-09T00:07:46.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 1 -- 172.21.15.73:0/1570290192 wait complete. 2024-03-09T00:07:46.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.661+0000 7faf34231740 20 client.15840 trim_cache size 0 max 16384 2024-03-09T00:07:46.662 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 msgr2=0x7faf180a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 0x7faf180a4fd0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7faf24009a60 tx=0x7faf24042040 comp rx=0 tx=0).stop 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 msgr2=0x7faf2c196700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 crypto rx=0x7faf1c00d950 tx=0x7faf1c00de10 comp rx=0 tx=0).stop 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 shutdown_connections 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faf2c197180 0x7faf2c1a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faf2c1054b0 0x7faf2c196c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faf180a2b10 0x7faf180a4fd0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 --2- 172.21.15.73:0/1098275506 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faf2c1003e0 0x7faf2c196700 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:46.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 >> 172.21.15.73:0/1098275506 conn(0x7faf2c0773a0 msgr2=0x7faf2c0fe190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:46.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 shutdown_connections 2024-03-09T00:07:46.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:46.662+0000 7faf3157d640 1 -- 172.21.15.73:0/1098275506 wait complete. 2024-03-09T00:07:47.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:47 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1536408828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:47.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:47 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1536408828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:47.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:47 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1536408828' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:07:47.672 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:07:47.672 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:07:47.672 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 -- 172.21.15.73:0/253432831 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39401054a0 msgr2=0x7f394010d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 --2- 172.21.15.73:0/253432831 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39401054a0 0x7f394010d070 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 crypto rx=0x7f3938009ca0 tx=0x7f393802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 -- 172.21.15.73:0/253432831 shutdown_connections 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 --2- 172.21.15.73:0/253432831 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39401054a0 0x7f394010d070 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 --2- 172.21.15.73:0/253432831 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3940100d80 0x7f3940104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 --2- 172.21.15.73:0/253432831 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401003d0 0x7f39401007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.788+0000 7f3947ca1640 1 -- 172.21.15.73:0/253432831 >> 172.21.15.73:0/253432831 conn(0x7f39400772f0 msgr2=0x7f3940077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 -- 172.21.15.73:0/253432831 shutdown_connections 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 -- 172.21.15.73:0/253432831 wait complete. 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 Processor -- start 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 -- start start 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 0x7f39401989e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.789+0000 7f3947ca1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401054a0 0x7f3940198f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3947ca1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 0x7f39401a3a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3947ca1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3940110560 con 0x7f3940199460 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3947ca1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401103e0 con 0x7f39401054a0 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3947ca1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401106e0 con 0x7f39401003d0 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3945a3d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 0x7f39401989e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3945a3d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 0x7f39401989e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:35050/0 (socket says 172.21.15.73:35050) 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f3945a3d640 1 -- 172.21.15.73:0/787055307 learned_addr learned my addr 172.21.15.73:0/787055307 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 0x7f39401a3a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394523c640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401054a0 0x7f3940198f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 -- 172.21.15.73:0/787055307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 msgr2=0x7f39401989e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 0x7f39401989e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 -- 172.21.15.73:0/787055307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401054a0 msgr2=0x7f3940198f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401054a0 0x7f3940198f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 -- 172.21.15.73:0/787055307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39401a41e0 con 0x7f3940199460 2024-03-09T00:07:47.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.790+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 0x7f39401a3a60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f393802c700 tx=0x7f3938032f40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.791+0000 7f3936ffd640 1 -- 172.21.15.73:0/787055307 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938041020 con 0x7f3940199460 2024-03-09T00:07:47.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.791+0000 7f3936ffd640 1 -- 172.21.15.73:0/787055307 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3938034cd0 con 0x7f3940199460 2024-03-09T00:07:47.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.791+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39401a4470 con 0x7f3940199460 2024-03-09T00:07:47.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.791+0000 7f3936ffd640 1 -- 172.21.15.73:0/787055307 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3938039040 con 0x7f3940199460 2024-03-09T00:07:47.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.791+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39401a4980 con 0x7f3940199460 2024-03-09T00:07:47.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.794+0000 7f3936ffd640 1 -- 172.21.15.73:0/787055307 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f3938044020 con 0x7f3940199460 2024-03-09T00:07:47.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.794+0000 7f394a522740 1 Processor -- start 2024-03-09T00:07:47.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.794+0000 7f394a522740 1 -- start start 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e4d80 0x55961a1e5160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1edd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a148b40 con 0x55961a1ee280 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a0f1d80 con 0x55961a1e56a0 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394a522740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a144be0 con 0x55961a1e4d80 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f3936ffd640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 0x7f392c0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47100/0 (socket says 172.21.15.73:47100) 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 -- 172.21.15.73:0/3866774050 learned_addr learned my addr 172.21.15.73:0/3866774050 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e4d80 0x55961a1e5160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 0x7f392c0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1edd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f3936ffd640 1 -- 172.21.15.73:0/787055307 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f39380f1070 con 0x7f3940199460 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 -- 172.21.15.73:0/3866774050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e4d80 msgr2=0x55961a1e5160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e4d80 0x55961a1e5160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 -- 172.21.15.73:0/3866774050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 msgr2=0x55961a1edd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1edd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f394523c640 1 -- 172.21.15.73:0/3866774050 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559619f257e0 con 0x55961a1ee280 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.795+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.796+0000 7f394523c640 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f393000bf80 tx=0x7f393000b6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a823784a40d15d0c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.796+0000 7f394623e640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 0x7f392c0a4fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39380ee420 tx=0x7f39380b2690 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.796+0000 7f392bfff640 1 -- 172.21.15.73:0/3866774050 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930004220 con 0x55961a1ee280 2024-03-09T00:07:47.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.796+0000 7f392bfff640 1 -- 172.21.15.73:0/3866774050 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39300043c0 con 0x55961a1ee280 2024-03-09T00:07:47.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.796+0000 7f392bfff640 1 -- 172.21.15.73:0/3866774050 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39300035f0 con 0x55961a1ee280 2024-03-09T00:07:47.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- 172.21.15.73:0/3866774050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 msgr2=0x55961a1f0670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 crypto rx=0x7f393000bf80 tx=0x7f393000b6e0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- 172.21.15.73:0/3866774050 shutdown_connections 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1f0670 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1edd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 --2- 172.21.15.73:0/3866774050 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e4d80 0x55961a1e5160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- 172.21.15.73:0/3866774050 >> 172.21.15.73:0/3866774050 conn(0x55961a1df4b0 msgr2=0x55961a1df890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- 172.21.15.73:0/3866774050 shutdown_connections 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- 172.21.15.73:0/3866774050 wait complete. 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.797+0000 7f394a522740 1 -- start start 2024-03-09T00:07:47.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 Processor -- start 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1edd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 0x55961a1eb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a1f1070 con 0x55961a1ee280 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a1051c0 con 0x55961a1e4d80 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55961a1f11f0 con 0x55961a1e56a0 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394a522740 10 client.0 _ll_register_callbacks cb 0x55961a0f47a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47110/0 (socket says 172.21.15.73:47110) 2024-03-09T00:07:47.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394523c640 1 -- 172.21.15.73:0/3374907326 learned_addr learned my addr 172.21.15.73:0/3374907326 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f394523c640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.798+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 0x55961a1eb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 -- 172.21.15.73:0/3374907326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 msgr2=0x55961a1eb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 0x55961a1eb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 -- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 msgr2=0x55961a1edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 -- 172.21.15.73:0/3374907326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559619f85f70 con 0x55961a1ee280 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 0x55961a1eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394523c640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3930004650 tx=0x7f3930004bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5ca014a7b0d9562c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f39297fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930031060 con 0x55961a1ee280 2024-03-09T00:07:47.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f39300043c0 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f39300053e0 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f39300055e0 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394a522740 10 client.15852 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559619e1d170 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.799+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f39300043c0 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f393001c070 con 0x55961a1ee280 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 10 client.15852 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 10 client.15852 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 20 client.15852 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a1fe6b0 0x55961a200af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 4 client.15852 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:47.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55961a11e960 con 0x55961a1fe6b0 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a1fe6b0 0x55961a200af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a1fe6b0 0x55961a200af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=206ea78f7fbb828c server_cookie=b43569136c38202d in_seq=0 out_seq=0 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.800+0000 7f39297fa640 10 client.15852 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f39297fa640 1 -- 172.21.15.73:0/3374907326 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55961a11e960 con 0x55961a1fe6b0 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f39297fa640 10 client.15852 handle_command_reply: tid=0 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 client.15852 shutdown 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 msgr2=0x55961a1eb7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 crypto rx=0x7f3930004650 tx=0x7f3930004bf0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 shutdown_connections 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a1fe6b0 0x55961a200af0 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1ee280 0x55961a1eb7d0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1e56a0 0x55961a1eb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 --2- 172.21.15.73:0/3374907326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 >> 172.21.15.73:0/3374907326 conn(0x55961a1df4b0 msgr2=0x55961a1e1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f39297fa640 0 client.15852 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 shutdown_connections 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.801+0000 7f394a522740 1 -- 172.21.15.73:0/3374907326 wait complete. 2024-03-09T00:07:47.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.802+0000 7f394a522740 20 client.15852 trim_cache size 0 max 16384 2024-03-09T00:07:47.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.830+0000 7f394a522740 1 Processor -- start 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 -- start start 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1f9ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a1edd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401ac600 con 0x55961a1e56a0 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401aebc0 con 0x55961a1e4d80 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394a522740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401acc00 con 0x55961a1ee280 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47122/0 (socket says 172.21.15.73:47122) 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 -- 172.21.15.73:0/3045532863 learned_addr learned my addr 172.21.15.73:0/3045532863 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394523c640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1f9ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394523c640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a1edd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 -- 172.21.15.73:0/3045532863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 msgr2=0x55961a1edd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a1edd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 -- 172.21.15.73:0/3045532863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 msgr2=0x55961a1f9ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1f9ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 -- 172.21.15.73:0/3045532863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55961a1e54d0 con 0x55961a1e56a0 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f394523c640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a1edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.831+0000 7f3945a3d640 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f393c010530 tx=0x7f393c005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6f536ed9e00d622 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f392b7fe640 1 -- 172.21.15.73:0/3045532863 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c005220 con 0x55961a1e56a0 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f392b7fe640 1 -- 172.21.15.73:0/3045532863 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f393c023030 con 0x55961a1e56a0 2024-03-09T00:07:47.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f392b7fe640 1 -- 172.21.15.73:0/3045532863 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f393c01f620 con 0x55961a1e56a0 2024-03-09T00:07:47.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 -- 172.21.15.73:0/3045532863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 msgr2=0x55961a1ed7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 crypto rx=0x7f393c010530 tx=0x7f393c005f90 comp rx=0 tx=0).stop 2024-03-09T00:07:47.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 -- 172.21.15.73:0/3045532863 shutdown_connections 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a1edd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e56a0 0x55961a1ed7c0 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 --2- 172.21.15.73:0/3045532863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e4d80 0x55961a1f9ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.832+0000 7f394a522740 1 -- 172.21.15.73:0/3045532863 >> 172.21.15.73:0/3045532863 conn(0x55961a1df4b0 msgr2=0x55961a205490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- 172.21.15.73:0/3045532863 shutdown_connections 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- 172.21.15.73:0/3045532863 wait complete. 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- start start 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 Processor -- start 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a24dc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401ad240 con 0x55961a1e4d80 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401ae560 con 0x55961a1e56a0 2024-03-09T00:07:47.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39401ad8a0 con 0x55961a1ee280 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394a522740 10 client.0 _ll_register_callbacks cb 0x7f393000ba30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394523c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47130/0 (socket says 172.21.15.73:47130) 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.833+0000 7f394523c640 1 -- 172.21.15.73:0/2486430479 learned_addr learned my addr 172.21.15.73:0/2486430479 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a24dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f3945a3d640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 -- 172.21.15.73:0/2486430479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 msgr2=0x55961a24dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a24dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 -- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 msgr2=0x55961a1f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 -- 172.21.15.73:0/2486430479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55961a24e240 con 0x55961a1e4d80 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f3945a3d640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394523c640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3930004620 tx=0x7f393001f040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=99fa267684002b3d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f39297fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3930031040 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3930010a30 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.834+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f393c023030 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3930004e10 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 10 client.15858 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55961a255590 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f393c023030 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3930045070 con 0x55961a1e4d80 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 10 client.15858 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 10 client.15858 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 20 client.15858 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a255c90 0x55961a2580b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 4 client.15858 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:07:47.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55961a10a080 con 0x55961a255c90 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.835+0000 7f3945a3d640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a255c90 0x55961a2580b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.836+0000 7f3945a3d640 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a255c90 0x55961a2580b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=908c5f9b9d829543 server_cookie=dbd9056c9a85ca82 in_seq=0 out_seq=0 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.836+0000 7f39297fa640 10 client.15858 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f39297fa640 1 -- 172.21.15.73:0/2486430479 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55961a10a080 con 0x55961a255c90 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f39297fa640 10 client.15858 handle_command_reply: tid=0 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 client.15858 shutdown 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 msgr2=0x55961a1f3710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 crypto rx=0x7f3930004620 tx=0x7f393001f040 comp rx=0 tx=0).stop 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 shutdown_connections 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55961a255c90 0x55961a2580b0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55961a1ee280 0x55961a24dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55961a1e56a0 0x55961a1f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f39297fa640 0 client.15858 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 --2- 172.21.15.73:0/2486430479 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55961a1e4d80 0x55961a1f3710 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 >> 172.21.15.73:0/2486430479 conn(0x55961a1df4b0 msgr2=0x55961a1e9830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 shutdown_connections 2024-03-09T00:07:47.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 1 -- 172.21.15.73:0/2486430479 wait complete. 2024-03-09T00:07:47.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.837+0000 7f394a522740 20 client.15858 trim_cache size 0 max 16384 2024-03-09T00:07:47.838 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:47.838 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:07:47.838 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "faa4de88-0d28-4b10-b198-e84f9b66f88d", 2024-03-09T00:07:47.838 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:07:47.838 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:47.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 msgr2=0x7f392c0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 0x7f392c0a4fd0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f39380ee420 tx=0x7f39380b2690 comp rx=0 tx=0).stop 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 msgr2=0x7f39401a3a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 0x7f39401a3a60 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 crypto rx=0x7f393802c700 tx=0x7f3938032f40 comp rx=0 tx=0).stop 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 shutdown_connections 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f392c0a2b10 0x7f392c0a4fd0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3940199460 0x7f39401a3a60 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39401054a0 0x7f3940198f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 --2- 172.21.15.73:0/787055307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39401003d0 0x7f39401989e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 >> 172.21.15.73:0/787055307 conn(0x7f39400772f0 msgr2=0x7f39400fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 shutdown_connections 2024-03-09T00:07:47.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.838+0000 7f3947ca1640 1 -- 172.21.15.73:0/787055307 wait complete. 2024-03-09T00:07:47.847 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:07:47.847 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:07:47.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/3550767014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e4068df0 msgr2=0x7fd4e410d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/3550767014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e4068df0 0x7fd4e410d750 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8009900 tx=0x7fd4d802bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/3550767014 shutdown_connections 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/3550767014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e410dc90 0x7fd4e4110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/3550767014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e4068df0 0x7fd4e410d750 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/3550767014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e40684d0 0x7fd4e40688b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/3550767014 >> 172.21.15.73:0/3550767014 conn(0x7fd4e40772f0 msgr2=0x7fd4e4077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/3550767014 shutdown_connections 2024-03-09T00:07:47.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.960+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/3550767014 wait complete. 2024-03-09T00:07:47.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 Processor -- start 2024-03-09T00:07:47.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 -- start start 2024-03-09T00:07:47.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 0x7fd4e419a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e410dc90 0x7fd4e419e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e4113870 con 0x7fd4e40684d0 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41136f0 con 0x7fd4e410dc90 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4ea0e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41139f0 con 0x7fd4e4068df0 2024-03-09T00:07:47.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47162/0 (socket says 172.21.15.73:47162) 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4e37fe640 1 -- 172.21.15.73:0/829251012 learned_addr learned my addr 172.21.15.73:0/829251012 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.961+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 0x7fd4e419a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e410dc90 0x7fd4e419e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/829251012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e410dc90 msgr2=0x7fd4e419e670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e410dc90 0x7fd4e419e670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 msgr2=0x7fd4e4199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/829251012 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4e419eda0 con 0x7fd4e4068df0 2024-03-09T00:07:47.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:47.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 0x7fd4e419a2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8036040 tx=0x7fd4d802ce20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e0ff9640 1 -- 172.21.15.73:0/829251012 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8041070 con 0x7fd4e4068df0 2024-03-09T00:07:47.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4e419f030 con 0x7fd4e4068df0 2024-03-09T00:07:47.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4e41a6920 con 0x7fd4e4068df0 2024-03-09T00:07:47.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e0ff9640 1 -- 172.21.15.73:0/829251012 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4d80054e0 con 0x7fd4e4068df0 2024-03-09T00:07:47.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.962+0000 7fd4e0ff9640 1 -- 172.21.15.73:0/829251012 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d8034440 con 0x7fd4e4068df0 2024-03-09T00:07:47.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 Processor -- start 2024-03-09T00:07:47.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 -- start start 2024-03-09T00:07:47.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af400720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af403050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af35ae00 con 0x55b0af400c60 2024-03-09T00:07:47.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af303b10 con 0x55b0af3f8080 2024-03-09T00:07:47.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.965+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af356ea0 con 0x55b0af3f7760 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:35110/0 (socket says 172.21.15.73:35110) 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 -- 172.21.15.73:0/1453014429 learned_addr learned my addr 172.21.15.73:0/1453014429 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af400720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 -- 172.21.15.73:0/1453014429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 msgr2=0x55b0af400720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af400720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e0ff9640 1 -- 172.21.15.73:0/829251012 <== mon.1 v2:172.21.15.179:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fd4d8044070 con 0x7fd4e4068df0 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 -- 172.21.15.73:0/1453014429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 msgr2=0x55b0af403050 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af403050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 -- 172.21.15.73:0/1453014429 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0af1397d0 con 0x55b0af3f7760 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e30 tx=0x7fd4dc010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d5f9f82744415de5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4c9ffb640 1 -- 172.21.15.73:0/1453014429 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc016070 con 0x55b0af3f7760 2024-03-09T00:07:47.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.966+0000 7fd4e0ff9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 0x7fd4cc0a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.967+0000 7fd4c9ffb640 1 -- 172.21.15.73:0/1453014429 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4dc004d20 con 0x55b0af3f7760 2024-03-09T00:07:47.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.967+0000 7fd4e0ff9640 1 -- 172.21.15.73:0/829251012 <== mon.1 v2:172.21.15.179:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd4d80f0f30 con 0x7fd4e4068df0 2024-03-09T00:07:47.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.967+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 0x7fd4cc0a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4c9ffb640 1 -- 172.21.15.73:0/1453014429 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4dc012850 con 0x55b0af3f7760 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 0x7fd4cc0a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55b0af3f7230 tx=0x7fd4d400b410 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/1453014429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 msgr2=0x55b0af3f7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fd4dc009e30 tx=0x7fd4dc010740 comp rx=0 tx=0).stop 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/1453014429 shutdown_connections 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af403050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af400720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/1453014429 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3f7b40 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/1453014429 >> 172.21.15.73:0/1453014429 conn(0x55b0af3f1e90 msgr2=0x55b0af3f2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/1453014429 shutdown_connections 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/1453014429 wait complete. 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.968+0000 7fd4ecd9b740 1 -- start start 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 Processor -- start 2024-03-09T00:07:47.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fd6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af400c60 0x55b0af3fdc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af403af0 con 0x55b0af3f7760 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af3170a0 con 0x55b0af3f8080 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b0af403c70 con 0x55b0af400c60 2024-03-09T00:07:47.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47186/0 (socket says 172.21.15.73:47186) 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 -- 172.21.15.73:0/2052367408 learned_addr learned my addr 172.21.15.73:0/2052367408 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4ecd9b740 10 client.0 _ll_register_callbacks cb 0x55b0af316a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fd6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af400c60 0x55b0af3fdc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 -- 172.21.15.73:0/2052367408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af400c60 msgr2=0x55b0af3fdc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af400c60 0x55b0af3fdc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 -- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 msgr2=0x55b0af3fd6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fd6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e37fe640 1 -- 172.21.15.73:0/2052367408 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0af06f3f0 con 0x55b0af3f7760 2024-03-09T00:07:47.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.969+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:47.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40109d0 tx=0x7fd4d4004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1eae20a3f90b882 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d402c060 con 0x55b0af3f7760 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4dc004d20 con 0x55b0af3f7760 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd4d4017030 con 0x55b0af3f7760 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d4033050 con 0x55b0af3f7760 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 10 client.15867 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:47.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55b0af02f0f0 con 0x55b0af3f7760 2024-03-09T00:07:47.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4dc004d20 con 0x55b0af3f7760 2024-03-09T00:07:47.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd4d4027cb0 con 0x55b0af3f7760 2024-03-09T00:07:47.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 10 client.15867 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:47.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 10 client.15867 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:47.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 20 client.15867 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.970+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af410780 0x55b0af412b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.971+0000 7fd4ecd9b740 4 client.15867 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.971+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b0af368340 con 0x55b0af410780 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.971+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af410780 0x55b0af412b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.971+0000 7fd4c2ffd640 10 client.15867 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.971+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af410780 0x55b0af412b70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b705d854e566d07f server_cookie=f53a72d74bad4679 in_seq=0 out_seq=0 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/2052367408 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b0af368340 con 0x55b0af410780 2024-03-09T00:07:47.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4c2ffd640 10 client.15867 handle_command_reply: tid=0 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 client.15867 shutdown 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 msgr2=0x55b0af4002f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 crypto rx=0x7fd4d40109d0 tx=0x7fd4d4004730 comp rx=0 tx=0).stop 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 shutdown_connections 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af410780 0x55b0af412b70 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af400c60 0x55b0af3fdc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fd6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4c2ffd640 0 client.15867 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:47.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2052367408 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af4002f0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 >> 172.21.15.73:0/2052367408 conn(0x55b0af3f1e90 msgr2=0x55b0af3f3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:47.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 shutdown_connections 2024-03-09T00:07:47.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.972+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2052367408 wait complete. 2024-03-09T00:07:47.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:47.973+0000 7fd4ecd9b740 20 client.15867 trim_cache size 0 max 16384 2024-03-09T00:07:47.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 Processor -- start 2024-03-09T00:07:47.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 -- start start 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3ffe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41ae890 con 0x55b0af400c60 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41b0e50 con 0x55b0af3f8080 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41aee90 con 0x55b0af3f7760 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e3fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47198/0 (socket says 172.21.15.73:47198) 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e3fff640 1 -- 172.21.15.73:0/2685096276 learned_addr learned my addr 172.21.15.73:0/2685096276 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3ffe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/2685096276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 msgr2=0x55b0af3ffe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3ffe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.002+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/2685096276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 msgr2=0x55b0af3fc500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4e2ffd640 1 -- 172.21.15.73:0/2685096276 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b0af417d10 con 0x55b0af3f8080 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fbfc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80ee220 tx=0x7fd4d80dd040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6cf5e3f252bab39c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4c97fa640 1 -- 172.21.15.73:0/2685096276 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d804b040 con 0x55b0af3f8080 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4c97fa640 1 -- 172.21.15.73:0/2685096276 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4d8005690 con 0x55b0af3f8080 2024-03-09T00:07:47.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.003+0000 7fd4c97fa640 1 -- 172.21.15.73:0/2685096276 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d80bb480 con 0x55b0af3f8080 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2685096276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 msgr2=0x55b0af3fbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fbfc0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4d80ee220 tx=0x7fd4d80dd040 comp rx=0 tx=0).stop 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2685096276 shutdown_connections 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af400c60 0x55b0af3fc500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af3f8080 0x55b0af3fbfc0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/2685096276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f7760 0x55b0af3ffe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2685096276 >> 172.21.15.73:0/2685096276 conn(0x55b0af3f1e90 msgr2=0x55b0af3fd6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2685096276 shutdown_connections 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/2685096276 wait complete. 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.004+0000 7fd4ecd9b740 1 -- start start 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 Processor -- start 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 0x55b0af3fab00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af400c60 0x55b0af3fb040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41af4d0 con 0x55b0af3f7760 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41b07f0 con 0x55b0af400c60 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4e41afb30 con 0x55b0af3f8080 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4ecd9b740 10 client.0 _ll_register_callbacks cb 0x7fd4d40106a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:48.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4e37fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:47202/0 (socket says 172.21.15.73:47202) 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4e37fe640 1 -- 172.21.15.73:0/896754880 learned_addr learned my addr 172.21.15.73:0/896754880 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 0x55b0af3fab00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.005+0000 7fd4e3fff640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af400c60 0x55b0af3fb040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 -- 172.21.15.73:0/896754880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 msgr2=0x55b0af3fab00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 0x55b0af3fab00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 -- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af400c60 msgr2=0x55b0af3fb040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af400c60 0x55b0af3fb040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 -- 172.21.15.73:0/896754880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b0af466e50 con 0x55b0af3f7760 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 0x55b0af3fab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4e37fe640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4020fd0 tx=0x7fd4d401c8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f6a804e8471833da server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4c2ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4d401cc60 con 0x55b0af3f7760 2024-03-09T00:07:48.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd4d401ce00 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55b0af362690 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4d4027410 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4ecd9b740 10 client.15873 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.006+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55b0af468140 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55b0af362690 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd4d40376e0 con 0x55b0af3f7760 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 10 client.15873 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:48.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 10 client.15873 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 20 client.15873 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af4688c0 0x55b0af46ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 4 client.15873 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b0af2e04e0 con 0x55b0af4688c0 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.007+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af4688c0 0x55b0af46ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4c2ffd640 10 client.15873 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4e2ffd640 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af4688c0 0x55b0af46ace0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ac3d8dbd4954ea4 server_cookie=58fe942fff6cc7c2 in_seq=0 out_seq=0 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4c2ffd640 1 -- 172.21.15.73:0/896754880 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55b0af2e04e0 con 0x55b0af4688c0 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4c2ffd640 10 client.15873 handle_command_reply: tid=0 2024-03-09T00:07:48.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 client.15873 shutdown 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 msgr2=0x55b0af3fa5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7fd4d4020fd0 tx=0x7fd4d401c8e0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 shutdown_connections 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b0af4688c0 0x55b0af46ace0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b0af400c60 0x55b0af3fb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4c2ffd640 0 client.15873 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b0af3f8080 0x55b0af3fab00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 --2- 172.21.15.73:0/896754880 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b0af3f7760 0x55b0af3fa5c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.008+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 >> 172.21.15.73:0/896754880 conn(0x55b0af3f1e90 msgr2=0x55b0af3fbe50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 shutdown_connections 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ecd9b740 1 -- 172.21.15.73:0/896754880 wait complete. 2024-03-09T00:07:48.009 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ecd9b740 20 client.15873 trim_cache size 0 max 16384 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "faa4de88-0d28-4b10-b198-e84f9b66f88d": { 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "faa4de88-0d28-4b10-b198-e84f9b66f88d", 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:07:48.010 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:07:48.011 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:07:48.011 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:07:48.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 msgr2=0x7fd4cc0a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 0x7fd4cc0a4fd0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x55b0af3f7230 tx=0x7fd4d400b410 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 msgr2=0x7fd4e419a2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.009+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 0x7fd4e419a2f0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd4d8036040 tx=0x7fd4d802ce20 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 shutdown_connections 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd4e410dc90 0x7fd4e419e670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd4e4068df0 0x7fd4e419a2f0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd4cc0a2b10 0x7fd4cc0a4fd0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 --2- 172.21.15.73:0/829251012 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd4e40684d0 0x7fd4e4199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 >> 172.21.15.73:0/829251012 conn(0x7fd4e40772f0 msgr2=0x7fd4e410f260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 shutdown_connections 2024-03-09T00:07:48.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:07:48.010+0000 7fd4ea0e9640 1 -- 172.21.15.73:0/829251012 wait complete. 2024-03-09T00:07:48.016 INFO:tasks.cephfs.filesystem:scrub status for tag:faa4de88-0d28-4b10-b198-e84f9b66f88d - {'path': '/', 'tag': 'faa4de88-0d28-4b10-b198-e84f9b66f88d', 'options': 'recursive,force'} 2024-03-09T00:07:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:48 smithi073 ceph-mon[32309]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-09T00:07:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:48 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:07:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:48 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:48 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:07:48.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:48 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:48.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:48 smithi102 ceph-mon[36106]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-09T00:07:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:48 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:07:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:48 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:48 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:07:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:48 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:48.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:48 smithi179 ceph-mon[36637]: pgmap v31: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 1.4 KiB/s wr, 4 op/s 2024-03-09T00:07:48.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:48 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:07:48.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:48 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:48.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:48 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:07:48.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:48 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:07:49.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:49 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:07:49.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:49 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:07:49.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:49 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:07:50.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:50 smithi073 ceph-mon[32309]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:50.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:50 smithi102 ceph-mon[36106]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:50.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:50 smithi179 ceph-mon[36637]: pgmap v32: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:52 smithi073 ceph-mon[32309]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:52.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:52 smithi102 ceph-mon[36106]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:52.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:52 smithi179 ceph-mon[36637]: pgmap v33: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:07:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:54 smithi073 ceph-mon[32309]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:54.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:54 smithi102 ceph-mon[36106]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:54.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:54 smithi179 ceph-mon[36637]: pgmap v34: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:56.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:56 smithi073 ceph-mon[32309]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:56.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:56 smithi102 ceph-mon[36106]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:56.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:56 smithi179 ceph-mon[36637]: pgmap v35: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:57 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:07:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:57 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:57 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:57 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:57 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:57 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:07:57.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:57 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:57 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:57 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:57 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:57 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:07:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:57 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:57 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:57 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:57 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:58 smithi073 ceph-mon[32309]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:58 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:07:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:58 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:07:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:58 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:58.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:58 smithi102 ceph-mon[36106]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:58 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:07:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:58 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:07:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:58 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:58.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:58 smithi179 ceph-mon[36637]: pgmap v36: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 255 B/s wr, 3 op/s 2024-03-09T00:07:58.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:58 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:07:58.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:58 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:07:58.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:58 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:07:58.927 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 -- 172.21.15.73:0/4131214781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 msgr2=0x7f0648105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.927 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 --2- 172.21.15.73:0/4131214781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f0648105570 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 crypto rx=0x7f0638009900 tx=0x7f063802bc30 comp rx=0 tx=0).stop 2024-03-09T00:07:58.927 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 -- 172.21.15.73:0/4131214781 shutdown_connections 2024-03-09T00:07:58.927 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 --2- 172.21.15.73:0/4131214781 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 0x7f064810d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.928 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 --2- 172.21.15.73:0/4131214781 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 0x7f06481008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.928 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 --2- 172.21.15.73:0/4131214781 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f0648105570 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.928 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 -- 172.21.15.73:0/4131214781 >> 172.21.15.73:0/4131214781 conn(0x7f0648077040 msgr2=0x7f0648077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.928 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 -- 172.21.15.73:0/4131214781 shutdown_connections 2024-03-09T00:07:58.928 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.932+0000 7f064de80640 1 -- 172.21.15.73:0/4131214781 wait complete. 2024-03-09T00:07:58.930 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.933+0000 7f064de80640 1 Processor -- start 2024-03-09T00:07:58.930 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.933+0000 7f064de80640 1 -- start start 2024-03-09T00:07:58.930 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 0x7f064819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 0x7f064819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481106a0 con 0x7f0648105190 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0648110520 con 0x7f0648100e00 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f064de80640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0648110820 con 0x7f0648100440 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0646ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 0x7f064819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43968/0 (socket says 172.21.15.73:43968) 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 -- 172.21.15.73:0/2835341568 learned_addr learned my addr 172.21.15.73:0/2835341568 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f06477fe640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 0x7f064819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 -- 172.21.15.73:0/2835341568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 msgr2=0x7f064819c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 0x7f064819c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 -- 172.21.15.73:0/2835341568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 msgr2=0x7f064819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.931 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 0x7f064819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.934+0000 7f0647fff640 1 -- 172.21.15.73:0/2835341568 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0648197040 con 0x7f0648105190 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f0647fff640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f064000cab0 tx=0x7f064000cf70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f0644ff9640 1 -- 172.21.15.73:0/2835341568 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f064000ae20 con 0x7f0648105190 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f0644ff9640 1 -- 172.21.15.73:0/2835341568 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f064000f930 con 0x7f0648105190 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0648197330 con 0x7f0648105190 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0648068970 con 0x7f0648105190 2024-03-09T00:07:58.932 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.935+0000 7f0644ff9640 1 -- 172.21.15.73:0/2835341568 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0640018b00 con 0x7f0648105190 2024-03-09T00:07:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.938+0000 7f0650a5c740 1 Processor -- start 2024-03-09T00:07:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0644ff9640 1 -- 172.21.15.73:0/2835341568 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f06400213f0 con 0x7f0648105190 2024-03-09T00:07:58.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 -- start start 2024-03-09T00:07:58.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3cd8690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cef400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3c3f5f0 con 0x55a0b3ce7790 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3be8930 con 0x55a0b3cd8bd0 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0650a5c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3c3b690 con 0x55a0b3cd8210 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.939+0000 7f0646ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43978/0 (socket says 172.21.15.73:43978) 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 -- 172.21.15.73:0/1158544130 learned_addr learned my addr 172.21.15.73:0/1158544130 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f06477fe640 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3cd8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cef400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 -- 172.21.15.73:0/1158544130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 msgr2=0x55a0b3cd8690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3cd8690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 -- 172.21.15.73:0/1158544130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 msgr2=0x55a0b3cef400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cef400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 -- 172.21.15.73:0/1158544130 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b3954210 con 0x55a0b3ce7790 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f063c013630 tx=0x7f063c009830 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7a253c64dcace9bb server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f062dffb640 1 -- 172.21.15.73:0/1158544130 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c004030 con 0x55a0b3ce7790 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f062dffb640 1 -- 172.21.15.73:0/1158544130 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063c0041d0 con 0x55a0b3ce7790 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.940+0000 7f062dffb640 1 -- 172.21.15.73:0/1158544130 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c00f7e0 con 0x55a0b3ce7790 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f0644ff9640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 0x7f06300a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f06477fe640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 0x7f06300a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f0644ff9640 1 -- 172.21.15.73:0/2835341568 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f06400cfaa0 con 0x7f0648105190 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f0650a5c740 1 -- 172.21.15.73:0/1158544130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 msgr2=0x55a0b3ce7b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 crypto rx=0x7f063c013630 tx=0x7f063c009830 comp rx=0 tx=0).stop 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f06477fe640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 0x7f06300a5070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f063802efd0 tx=0x7f063802e040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.941+0000 7f0650a5c740 1 -- 172.21.15.73:0/1158544130 shutdown_connections 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cef400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3cd8690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1158544130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ce7790 0x55a0b3ce7b70 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 -- 172.21.15.73:0/1158544130 >> 172.21.15.73:0/1158544130 conn(0x55a0b3ca25b0 msgr2=0x55a0b3cd7100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.943 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 -- 172.21.15.73:0/1158544130 shutdown_connections 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 -- 172.21.15.73:0/1158544130 wait complete. 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 -- start start 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.942+0000 7f0650a5c740 1 Processor -- start 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3ceca40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cecf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43984/0 (socket says 172.21.15.73:43984) 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0647fff640 1 -- 172.21.15.73:0/1563738853 learned_addr learned my addr 172.21.15.73:0/1563738853 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3cefaf0 con 0x55a0b3ced4c0 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3bfbd90 con 0x55a0b3cd8bd0 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0b3cefc70 con 0x55a0b3cd8210 2024-03-09T00:07:58.944 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f06477fe640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3ceca40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cecf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.943+0000 7f0650a5c740 10 client.0 _ll_register_callbacks cb 0x55a0b3953a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 -- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 msgr2=0x55a0b3cecf80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cecf80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 -- 172.21.15.73:0/1563738853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 msgr2=0x55a0b3cf8f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0b3a20620 con 0x55a0b3cd8210 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f0647fff640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f06477fe640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3ceca40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f063802c170 tx=0x7f063803d040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=249c21a3fa594e42 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f0626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638055060 con 0x55a0b3cd8210 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0638004ad0 con 0x55a0b3cd8210 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.944+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f063c0041d0 con 0x55a0b3cd8210 2024-03-09T00:07:58.945 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063805c050 con 0x55a0b3cd8210 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 10 client.25393 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0b3cfc300 con 0x55a0b3cd8210 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063c0041d0 con 0x55a0b3cd8210 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f063804f030 con 0x55a0b3cd8210 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 10 client.25393 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 10 client.25393 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 20 client.25393 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.945+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3cfdba0 0x55a0b3cfffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.946+0000 7f0650a5c740 4 client.25393 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.946+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0b3c4cb30 con 0x55a0b3cfdba0 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.946+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3cfdba0 0x55a0b3cfffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.946+0000 7f0626ffd640 10 client.25393 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:58.946 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.946+0000 7f0646ffd640 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3cfdba0 0x55a0b3cfffb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6092a6eb75d02c3b server_cookie=2e59d9ab605c60eb in_seq=0 out_seq=0 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.947+0000 7f0626ffd640 1 -- 172.21.15.73:0/1563738853 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0b3c4cb30 con 0x55a0b3cfdba0 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.947+0000 7f0626ffd640 10 client.25393 handle_command_reply: tid=0 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.947+0000 7f0650a5c740 1 client.25393 shutdown 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 msgr2=0x55a0b3ceca40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3ceca40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f063802c170 tx=0x7f063803d040 comp rx=0 tx=0).stop 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 shutdown_connections 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3cfdba0 0x55a0b3cfffb0 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ced4c0 0x55a0b3cf8f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cecf80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 --2- 172.21.15.73:0/1563738853 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8210 0x55a0b3ceca40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 >> 172.21.15.73:0/1563738853 conn(0x55a0b3ca25b0 msgr2=0x55a0b3cd4e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0626ffd640 0 client.25393 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 shutdown_connections 2024-03-09T00:07:58.947 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 1 -- 172.21.15.73:0/1563738853 wait complete. 2024-03-09T00:07:58.948 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.948+0000 7f0650a5c740 20 client.25393 trim_cache size 0 max 16384 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 Processor -- start 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 -- start start 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3ca25b0 0x55a0b3d04910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf8960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481aa390 con 0x55a0b3cd8bd0 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481ac950 con 0x55a0b3cd8210 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0650a5c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481aa990 con 0x55a0b3ca25b0 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0646ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf8960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0647fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43992/0 (socket says 172.21.15.73:43992) 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f0647fff640 1 -- 172.21.15.73:0/583003578 learned_addr learned my addr 172.21.15.73:0/583003578 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.974+0000 7f06477fe640 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3ca25b0 0x55a0b3d04910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 -- 172.21.15.73:0/583003578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3ca25b0 msgr2=0x55a0b3d04910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3ca25b0 0x55a0b3d04910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 -- 172.21.15.73:0/583003578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 msgr2=0x55a0b3cf8ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 -- 172.21.15.73:0/583003578 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0b3cd1590 con 0x55a0b3cd8210 2024-03-09T00:07:58.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0647fff640 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:07:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f0646ffd640 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf8960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55a0b3cd6b40 tx=0x7f063c003e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8d6ee8fc234aa976 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f062d7fa640 1 -- 172.21.15.73:0/583003578 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c030070 con 0x55a0b3cd8210 2024-03-09T00:07:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f062d7fa640 1 -- 172.21.15.73:0/583003578 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063c02a840 con 0x55a0b3cd8210 2024-03-09T00:07:58.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.975+0000 7f062d7fa640 1 -- 172.21.15.73:0/583003578 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063c003850 con 0x55a0b3cd8210 2024-03-09T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 -- 172.21.15.73:0/583003578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 msgr2=0x55a0b3cf8960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf8960 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x55a0b3cd6b40 tx=0x7f063c003e70 comp rx=0 tx=0).stop 2024-03-09T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 -- 172.21.15.73:0/583003578 shutdown_connections 2024-03-09T00:07:58.978 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf8ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf8960 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 --2- 172.21.15.73:0/583003578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3ca25b0 0x55a0b3d04910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 -- 172.21.15.73:0/583003578 >> 172.21.15.73:0/583003578 conn(0x55a0b3ce71c0 msgr2=0x55a0b3cd6e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 -- 172.21.15.73:0/583003578 shutdown_connections 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.976+0000 7f0650a5c740 1 -- 172.21.15.73:0/583003578 wait complete. 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 -- start start 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 Processor -- start 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf24b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf29f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481aafd0 con 0x55a0b3ca25b0 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481ac2f0 con 0x55a0b3cd8210 2024-03-09T00:07:58.979 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f0650a5c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06481ab630 con 0x55a0b3cd8bd0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.977+0000 7f06477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43996/0 (socket says 172.21.15.73:43996) 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 -- 172.21.15.73:0/3902133870 learned_addr learned my addr 172.21.15.73:0/3902133870 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0650a5c740 10 client.0 _ll_register_callbacks cb 0x7f063c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0646ffd640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf24b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0647fff640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf29f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 -- 172.21.15.73:0/3902133870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8bd0 msgr2=0x55a0b3cf29f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf29f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 -- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 msgr2=0x55a0b3cf24b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf24b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 -- 172.21.15.73:0/3902133870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0b3cf3080 con 0x55a0b3ca25b0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f06477fe640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0638003910 tx=0x7f0638005bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5878daed6b34f93 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0626ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:07:58.980 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0638065060 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f06380048d0 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.978+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f063c02a840 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063806c050 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 10 client.15891 fetch_fsmap learned FSMap version 20 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0b3d54f50 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0638004ad0 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f063804f050 con 0x55a0b3ca25b0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 10 client.15891 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 10 client.15891 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 20 client.15891 populate_metadata read hostname 'smithi073' 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3ced4c0 0x55a0b3d57660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 4 client.15891 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.979+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x7f063c02a840 con 0x55a0b3ced4c0 2024-03-09T00:07:58.981 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.980+0000 7f0646ffd640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3ced4c0 0x55a0b3d57660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:07:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.981+0000 7f0646ffd640 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3ced4c0 0x55a0b3d57660 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b804707f76dc8d4d server_cookie=15becf85ed59e6b1 in_seq=0 out_seq=0 2024-03-09T00:07:58.982 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.981+0000 7f0626ffd640 10 client.15891 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0626ffd640 1 -- 172.21.15.73:0/3902133870 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x7f063c02a840 con 0x55a0b3ced4c0 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0626ffd640 10 client.15891 handle_command_reply: tid=0 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 client.15891 shutdown 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 msgr2=0x55a0b3cf4110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 crypto rx=0x7f0638003910 tx=0x7f0638005bc0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 shutdown_connections 2024-03-09T00:07:58.983 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0b3ced4c0 0x55a0b3d57660 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0b3cd8bd0 0x55a0b3cf29f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0b3cd8210 0x55a0b3cf24b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 --2- 172.21.15.73:0/3902133870 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0b3ca25b0 0x55a0b3cf4110 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 >> 172.21.15.73:0/3902133870 conn(0x55a0b3ce71c0 msgr2=0x55a0b3d04910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0626ffd640 0 client.15891 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 shutdown_connections 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 1 -- 172.21.15.73:0/3902133870 wait complete. 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.985+0000 7f0650a5c740 20 client.15891 trim_cache size 0 max 16384 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:07:58.984 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:07:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:07:58.985 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 msgr2=0x7f06300a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 0x7f06300a5070 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f063802efd0 tx=0x7f063802e040 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 msgr2=0x7f06481968c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 crypto rx=0x7f064000cab0 tx=0x7f064000cf70 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 shutdown_connections 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06300a2bb0 0x7f06300a5070 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0648105190 0x7f06481968c0 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0648100e00 0x7f064819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 --2- 172.21.15.73:0/2835341568 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0648100440 0x7f064819c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 >> 172.21.15.73:0/2835341568 conn(0x7f0648077040 msgr2=0x7f06480fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 shutdown_connections 2024-03-09T00:07:58.986 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:07:58.986+0000 7f064de80640 1 -- 172.21.15.73:0/2835341568 wait complete. 2024-03-09T00:07:59.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:07:59 smithi102 ceph-mon[36106]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:07:59.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:07:59 smithi179 ceph-mon[36637]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:07:59.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:07:59 smithi073 ceph-mon[32309]: pgmap v37: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:08:02.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:02 smithi179 ceph-mon[36637]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:08:02.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:02 smithi073 ceph-mon[32309]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:08:02.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:02 smithi102 ceph-mon[36106]: pgmap v38: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 2.0 KiB/s rd, 2 op/s 2024-03-09T00:08:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:04 smithi179 ceph-mon[36637]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-09T00:08:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:04 smithi073 ceph-mon[32309]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-09T00:08:04.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:04 smithi102 ceph-mon[36106]: pgmap v39: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 22 KiB/s wr, 18 op/s 2024-03-09T00:08:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:06 smithi179 ceph-mon[36637]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:08:06.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:06 smithi073 ceph-mon[32309]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:08:06.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:06 smithi102 ceph-mon[36106]: pgmap v40: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:08:08.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:08 smithi179 ceph-mon[36637]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:08.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:08 smithi073 ceph-mon[32309]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:08.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:08 smithi102 ceph-mon[36106]: pgmap v41: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:09.242 INFO:tasks.workunit.client.0.smithi073.stderr:Note: switching to '7d63482aa89628d733e298984a762389b625c761'. 2024-03-09T00:08:09.242 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.242 INFO:tasks.workunit.client.0.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr: git switch -c 2024-03-09T00:08:09.243 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.244 INFO:tasks.workunit.client.0.smithi073.stderr:Or undo this operation with: 2024-03-09T00:08:09.244 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.244 INFO:tasks.workunit.client.0.smithi073.stderr: git switch - 2024-03-09T00:08:09.244 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.244 INFO:tasks.workunit.client.0.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T00:08:09.245 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:09.245 INFO:tasks.workunit.client.0.smithi073.stderr:HEAD is now at 7d63482aa89 Merge remote-tracking branch 'rishabh/fsshell-deprecation' into rishabh-2024mar8 2024-03-09T00:08:09.249 DEBUG:teuthology.orchestra.run.smithi073:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-09T00:08:09.324 INFO:tasks.workunit.client.0.smithi073.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-09T00:08:09.327 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T00:08:09.327 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-09T00:08:10.050 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-09T00:08:10.120 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-09T00:08:10.166 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-09T00:08:10.167 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T00:08:10.167 INFO:tasks.workunit.client.0.smithi073.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-09T00:08:10.201 INFO:tasks.workunit.client.0.smithi073.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-09T00:08:10.215 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:08:10.215 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-09T00:08:10.273 INFO:tasks.workunit:Running workunits matching suites/fsx.sh on client.0... 2024-03-09T00:08:10.276 INFO:tasks.workunit:Running workunit suites/fsx.sh... 2024-03-09T00:08:10.277 DEBUG:teuthology.orchestra.run.smithi073:workunit test suites/fsx.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh 2024-03-09T00:08:10.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:10 smithi179 ceph-mon[36637]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:10.336 INFO:tasks.workunit.client.0.smithi073.stderr:+ set -e 2024-03-09T00:08:10.336 INFO:tasks.workunit.client.0.smithi073.stderr:+ git clone https://git.ceph.com/xfstests-dev.git 2024-03-09T00:08:10.338 INFO:tasks.workunit.client.0.smithi073.stderr:Cloning into 'xfstests-dev'... 2024-03-09T00:08:10.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:10 smithi073 ceph-mon[32309]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:10.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:10 smithi102 ceph-mon[36106]: pgmap v42: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 -- 172.21.15.73:0/2495211472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 msgr2=0x7ffb301077c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 --2- 172.21.15.73:0/2495211472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb301077c0 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 crypto rx=0x7ffb2c009900 tx=0x7ffb2c02bcd0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 -- 172.21.15.73:0/2495211472 shutdown_connections 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 --2- 172.21.15.73:0/2495211472 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30069310 0x7ffb3010e180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 --2- 172.21.15.73:0/2495211472 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30068970 0x7ffb30068dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 --2- 172.21.15.73:0/2495211472 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb301077c0 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 -- 172.21.15.73:0/2495211472 >> 172.21.15.73:0/2495211472 conn(0x7ffb30077040 msgr2=0x7ffb30077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 -- 172.21.15.73:0/2495211472 shutdown_connections 2024-03-09T00:08:12.082 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.080+0000 7ffb377a9640 1 -- 172.21.15.73:0/2495211472 wait complete. 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 Processor -- start 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 -- start start 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30068970 0x7ffb30100510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30069310 0x7ffb30100a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb30111660 con 0x7ffb301073e0 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301114e0 con 0x7ffb30068970 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb377a9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301117e0 con 0x7ffb30069310 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56824/0 (socket says 172.21.15.73:56824) 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 -- 172.21.15.73:0/478635123 learned_addr learned my addr 172.21.15.73:0/478635123 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb34d44640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30069310 0x7ffb30100a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35545640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30068970 0x7ffb30100510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.083 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 -- 172.21.15.73:0/478635123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30069310 msgr2=0x7ffb30100a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30069310 0x7ffb30100a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 -- 172.21.15.73:0/478635123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30068970 msgr2=0x7ffb30100510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30068970 0x7ffb30100510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.081+0000 7ffb35d46640 1 -- 172.21.15.73:0/478635123 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb301a5fa0 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb35d46640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800e960 tx=0x7ffb2800ee20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb267fc640 1 -- 172.21.15.73:0/478635123 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb28004d20 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb267fc640 1 -- 172.21.15.73:0/478635123 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb28005020 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb301a6290 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb267fc640 1 -- 172.21.15.73:0/478635123 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb280116d0 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.082+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb301a67e0 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.085+0000 7ffb267fc640 1 -- 172.21.15.73:0/478635123 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ffb28005560 con 0x7ffb301073e0 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.085+0000 7ffb3a385740 1 Processor -- start 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.085+0000 7ffb3a385740 1 -- start start 2024-03-09T00:08:12.084 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.085+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28ffd40 0x55a0d28fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.085+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d28ff450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb3a385740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d28635f0 con 0x55a0d2908550 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb3a385740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d280c930 con 0x55a0d28fefd0 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb3a385740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d285f690 con 0x55a0d28ffd40 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56828/0 (socket says 172.21.15.73:56828) 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 -- 172.21.15.73:0/1218224559 learned_addr learned my addr 172.21.15.73:0/1218224559 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb267fc640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 0x7ffb1c0a5070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28ffd40 0x55a0d28fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb35545640 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d28ff450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb267fc640 1 -- 172.21.15.73:0/478635123 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ffb280cf5b0 con 0x7ffb301073e0 2024-03-09T00:08:12.085 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 -- 172.21.15.73:0/1218224559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28ffd40 msgr2=0x55a0d28fea90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28ffd40 0x55a0d28fea90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 -- 172.21.15.73:0/1218224559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 msgr2=0x55a0d28ff450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d28ff450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb35d46640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 0x7ffb1c0a5070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 -- 172.21.15.73:0/1218224559 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0d2578210 con 0x55a0d2908550 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.086+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ffb200127a0 tx=0x7ffb2000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=663f0f60d31a8484 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.087+0000 7ffb1b7fe640 1 -- 172.21.15.73:0/1218224559 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20012d20 con 0x55a0d2908550 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.087+0000 7ffb1b7fe640 1 -- 172.21.15.73:0/1218224559 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb20005020 con 0x55a0d2908550 2024-03-09T00:08:12.086 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.087+0000 7ffb1b7fe640 1 -- 172.21.15.73:0/1218224559 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb20004b20 con 0x55a0d2908550 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- 172.21.15.73:0/1218224559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 msgr2=0x55a0d290a940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb35d46640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 0x7ffb1c0a5070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cc850 tx=0x7ffb280cf8b0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 crypto rx=0x7ffb200127a0 tx=0x7ffb2000b7c0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- 172.21.15.73:0/1218224559 shutdown_connections 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d2908550 0x55a0d290a940 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.091 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d28ff450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1218224559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28ffd40 0x55a0d28fea90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- 172.21.15.73:0/1218224559 >> 172.21.15.73:0/1218224559 conn(0x55a0d28c65b0 msgr2=0x55a0d28f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- 172.21.15.73:0/1218224559 shutdown_connections 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- 172.21.15.73:0/1218224559 wait complete. 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 -- start start 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.088+0000 7ffb3a385740 1 Processor -- start 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d2905b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d2906060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d290b280 con 0x55a0d28ffd40 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d281fd90 con 0x55a0d28fefd0 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a0d290b400 con 0x55a0d2908550 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb3a385740 10 client.0 _ll_register_callbacks cb 0x55a0d2577a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:12.092 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb35545640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d2905b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:40184/0 (socket says 172.21.15.73:40184) 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb34d44640 1 -- 172.21.15.73:0/1307064570 learned_addr learned my addr 172.21.15.73:0/1307064570 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.089+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d2906060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 -- 172.21.15.73:0/1307064570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 msgr2=0x55a0d2906060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d2906060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 -- 172.21.15.73:0/1307064570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 msgr2=0x55a0d2905b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d2905b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 -- 172.21.15.73:0/1307064570 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0d2644620 con 0x55a0d28fefd0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb35545640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d2905b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb34d44640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb20001eb0 tx=0x7ffb2001b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c37ef17df848a171 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb200048a0 con 0x55a0d28fefd0 2024-03-09T00:08:12.093 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb20005020 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ffb20004a40 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb20033050 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb3a385740 10 client.25337 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.090+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0d29175c0 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb20005020 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb2001f420 con 0x55a0d28fefd0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 10 client.25337 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 10 client.25337 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 20 client.25337 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d291ad00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 4 client.25337 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a0d2870b30 con 0x55a0d2918910 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.091+0000 7ffb35545640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d291ad00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.094 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb18ff9640 10 client.25337 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb35545640 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d291ad00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50c07f58d8d0b428 server_cookie=7bc016cd9133c8bb in_seq=0 out_seq=0 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb18ff9640 1 -- 172.21.15.73:0/1307064570 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55a0d2870b30 con 0x55a0d2918910 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb18ff9640 10 client.25337 handle_command_reply: tid=0 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb3a385740 1 client.25337 shutdown 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 msgr2=0x55a0d29055e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7ffb20001eb0 tx=0x7ffb2001b040 comp rx=0 tx=0).stop 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 shutdown_connections 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.092+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d291ad00 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d2906060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb18ff9640 0 client.25337 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d2905b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 --2- 172.21.15.73:0/1307064570 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d29055e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 >> 172.21.15.73:0/1307064570 conn(0x55a0d28c65b0 msgr2=0x55a0d28fba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 shutdown_connections 2024-03-09T00:08:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 1 -- 172.21.15.73:0/1307064570 wait complete. 2024-03-09T00:08:12.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.093+0000 7ffb3a385740 20 client.25337 trim_cache size 0 max 16384 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 Processor -- start 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 -- start start 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28c65b0 0x55a0d2907d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d2904490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301ae900 con 0x55a0d28ffd40 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301b0ec0 con 0x55a0d28fefd0 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb3a385740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301aef00 con 0x55a0d28c65b0 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56858/0 (socket says 172.21.15.73:56858) 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 -- 172.21.15.73:0/500157581 learned_addr learned my addr 172.21.15.73:0/500157581 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28c65b0 0x55a0d2907d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.103 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb35545640 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d2904490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 -- 172.21.15.73:0/500157581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28c65b0 msgr2=0x55a0d2907d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28c65b0 0x55a0d2907d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 -- 172.21.15.73:0/500157581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 msgr2=0x55a0d2904490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d2904490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.107+0000 7ffb34d44640 1 -- 172.21.15.73:0/500157581 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a0d28fe2a0 con 0x55a0d28ffd40 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.108+0000 7ffb34d44640 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffb20000c00 tx=0x7ffb2000b7c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff575596800e990e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.108+0000 7ffb1affd640 1 -- 172.21.15.73:0/500157581 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000bcd0 con 0x55a0d28ffd40 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.108+0000 7ffb1affd640 1 -- 172.21.15.73:0/500157581 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb20016040 con 0x55a0d28ffd40 2024-03-09T00:08:12.104 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.108+0000 7ffb1affd640 1 -- 172.21.15.73:0/500157581 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2001f420 con 0x55a0d28ffd40 2024-03-09T00:08:12.107 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- 172.21.15.73:0/500157581 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 msgr2=0x55a0d290f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 crypto rx=0x7ffb20000c00 tx=0x7ffb2000b7c0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- 172.21.15.73:0/500157581 shutdown_connections 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28ffd40 0x55a0d290f670 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28fefd0 0x55a0d2904490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 --2- 172.21.15.73:0/500157581 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d28c65b0 0x55a0d2907d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- 172.21.15.73:0/500157581 >> 172.21.15.73:0/500157581 conn(0x55a0d290ab20 msgr2=0x55a0d2903d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- 172.21.15.73:0/500157581 shutdown_connections 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- 172.21.15.73:0/500157581 wait complete. 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 -- start start 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.109+0000 7ffb3a385740 1 Processor -- start 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 0x55a0d2902e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.108 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d296ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301af540 con 0x55a0d28fefd0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301b0860 con 0x55a0d28c65b0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb301afba0 con 0x55a0d2908550 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb35545640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 0x55a0d2902e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb3a385740 10 client.0 _ll_register_callbacks cb 0x7ffb2000bab0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:40192/0 (socket says 172.21.15.73:40192) 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 -- 172.21.15.73:0/680589633 learned_addr learned my addr 172.21.15.73:0/680589633 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d296ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 -- 172.21.15.73:0/680589633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 msgr2=0x55a0d296ec00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d296ec00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 -- 172.21.15.73:0/680589633 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 msgr2=0x55a0d2902e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 0x55a0d2902e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 -- 172.21.15.73:0/680589633 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a0d2914690 con 0x55a0d28c65b0 2024-03-09T00:08:12.109 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb35545640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 0x55a0d2902e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb34d44640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb2001f870 tx=0x7ffb2002a040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=148d007b9801acee server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.110+0000 7ffb18ff9640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb2000bcd0 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ffb20010740 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb20016040 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb20005610 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 10 client.25343 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55a0d29701f0 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb20016040 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ffb20004a00 con 0x55a0d28c65b0 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 10 client.25343 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 10 client.25343 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:12.110 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 20 client.25343 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d29727c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 4 client.25343 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55a0d286ae80 con 0x55a0d2918910 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.111+0000 7ffb35545640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d29727c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.112+0000 7ffb18ff9640 10 client.25343 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:12.111 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.112+0000 7ffb35545640 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d29727c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44a76bfd84a93ccc server_cookie=d1b4918f84b1e993 in_seq=0 out_seq=0 2024-03-09T00:08:12.142 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.146+0000 7ffb18ff9640 1 -- 172.21.15.73:0/680589633 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55a0d286ae80 con 0x55a0d2918910 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.146+0000 7ffb18ff9640 10 client.25343 handle_command_reply: tid=0 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 client.25343 shutdown 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 msgr2=0x55a0d2902950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7ffb2001f870 tx=0x7ffb2002a040 comp rx=0 tx=0).stop 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 shutdown_connections 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55a0d2918910 0x55a0d29727c0 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55a0d2908550 0x55a0d296ec00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb18ff9640 0 client.25343 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55a0d28fefd0 0x55a0d2902e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 --2- 172.21.15.73:0/680589633 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55a0d28c65b0 0x55a0d2902950 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 >> 172.21.15.73:0/680589633 conn(0x55a0d290ab20 msgr2=0x55a0d2904610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.143 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 shutdown_connections 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 1 -- 172.21.15.73:0/680589633 wait complete. 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.147+0000 7ffb3a385740 20 client.25343 trim_cache size 0 max 16384 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:08:12.144 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 msgr2=0x7ffb1c0a5070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 0x7ffb1c0a5070 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7ffb280cc850 tx=0x7ffb280cf8b0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 msgr2=0x7ffb30100f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 crypto rx=0x7ffb2800e960 tx=0x7ffb2800ee20 comp rx=0 tx=0).stop 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 shutdown_connections 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffb1c0a2bb0 0x7ffb1c0a5070 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.145 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffb301073e0 0x7ffb30100f90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffb30069310 0x7ffb30100a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 --2- 172.21.15.73:0/478635123 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffb30068970 0x7ffb30100510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 >> 172.21.15.73:0/478635123 conn(0x7ffb30077040 msgr2=0x7ffb300ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 shutdown_connections 2024-03-09T00:08:12.146 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:12.148+0000 7ffb377a9640 1 -- 172.21.15.73:0/478635123 wait complete. 2024-03-09T00:08:12.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:12 smithi179 ceph-mon[36637]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:12.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:12 smithi073 ceph-mon[32309]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:12.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:12 smithi102 ceph-mon[36106]: pgmap v43: 129 pgs: 129 active+clean; 1.4 MiB data, 394 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:08:14.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:14 smithi179 ceph-mon[36637]: pgmap v44: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-03-09T00:08:14.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:14 smithi073 ceph-mon[32309]: pgmap v44: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-03-09T00:08:14.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:14 smithi102 ceph-mon[36106]: pgmap v44: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 23 KiB/s rd, 38 KiB/s wr, 19 op/s 2024-03-09T00:08:16.311 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:16 smithi102 ceph-mon[36106]: pgmap v45: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 3 op/s 2024-03-09T00:08:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:16 smithi179 ceph-mon[36637]: pgmap v45: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 3 op/s 2024-03-09T00:08:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:16 smithi073 ceph-mon[32309]: pgmap v45: 129 pgs: 129 active+clean; 1.5 MiB data, 395 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 15 KiB/s wr, 3 op/s 2024-03-09T00:08:18.017 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:08:18.017 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:08:18.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 -- 172.21.15.73:0/4158285478 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 msgr2=0x7efdfc10e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 --2- 172.21.15.73:0/4158285478 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc10e170 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 crypto rx=0x7efdf4009cd0 tx=0x7efdf402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 -- 172.21.15.73:0/4158285478 shutdown_connections 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 --2- 172.21.15.73:0/4158285478 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc10e170 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 --2- 172.21.15.73:0/4158285478 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc0ffbb0 0x7efdfc100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 --2- 172.21.15.73:0/4158285478 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc1074f0 0x7efdfc1078d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 -- 172.21.15.73:0/4158285478 >> 172.21.15.73:0/4158285478 conn(0x7efdfc075530 msgr2=0x7efdfc075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 -- 172.21.15.73:0/4158285478 shutdown_connections 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.145+0000 7efe02905640 1 -- 172.21.15.73:0/4158285478 wait complete. 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 Processor -- start 2024-03-09T00:08:18.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 -- start start 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc0ffbb0 0x7efdfc19d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc1074f0 0x7efdfc197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1116f0 con 0x7efdfc100570 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc111570 con 0x7efdfc0ffbb0 2024-03-09T00:08:18.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efe02905640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc111870 con 0x7efdfc1074f0 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.146+0000 7efdfb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56894/0 (socket says 172.21.15.73:56894) 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 -- 172.21.15.73:0/4211056040 learned_addr learned my addr 172.21.15.73:0/4211056040 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfbfff640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc0ffbb0 0x7efdfc19d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efe00ea2640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc1074f0 0x7efdfc197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 -- 172.21.15.73:0/4211056040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc1074f0 msgr2=0x7efdfc197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc1074f0 0x7efdfc197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 -- 172.21.15.73:0/4211056040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc0ffbb0 msgr2=0x7efdfc19d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc0ffbb0 0x7efdfc19d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 -- 172.21.15.73:0/4211056040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdfc198110 con 0x7efdfc100570 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7efdec00e9f0 tx=0x7efdec00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdf97fa640 1 -- 172.21.15.73:0/4211056040 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec00cd80 con 0x7efdfc100570 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdfc198400 con 0x7efdfc100570 2024-03-09T00:08:18.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdf97fa640 1 -- 172.21.15.73:0/4211056040 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdec005500 con 0x7efdfc100570 2024-03-09T00:08:18.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdfc1a46a0 con 0x7efdfc100570 2024-03-09T00:08:18.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.147+0000 7efdf97fa640 1 -- 172.21.15.73:0/4211056040 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdec018630 con 0x7efdfc100570 2024-03-09T00:08:18.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efdf97fa640 1 -- 172.21.15.73:0/4211056040 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7efdec0046c0 con 0x7efdfc100570 2024-03-09T00:08:18.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 Processor -- start 2024-03-09T00:08:18.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 -- start start 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4b5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4bec60 0x5623eb4c1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb418e00 con 0x5623eb4b6080 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb3c1b10 con 0x5623eb4b5760 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efe055b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb414ea0 con 0x5623eb4bec60 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.150+0000 7efdf97fa640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 0x7efde40a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efe00ea2640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 0x7efde40a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56902/0 (socket says 172.21.15.73:56902) 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 -- 172.21.15.73:0/3318395910 learned_addr learned my addr 172.21.15.73:0/3318395910 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdf97fa640 1 -- 172.21.15.73:0/4211056040 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efdec0cf9c0 con 0x7efdfc100570 2024-03-09T00:08:18.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efe00ea2640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4b5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4bec60 0x5623eb4c1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 -- 172.21.15.73:0/3318395910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4bec60 msgr2=0x5623eb4c1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4bec60 0x5623eb4c1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 -- 172.21.15.73:0/3318395910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 msgr2=0x5623eb4b5b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4b5b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 -- 172.21.15.73:0/3318395910 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623eb1f77d0 con 0x5623eb4b6080 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efe00ea2640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efdfbfff640 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efdf000bcb0 tx=0x7efdf00107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7418ab162ee6982 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efe00ea2640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 0x7efde40a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efdf4005cc0 tx=0x7efdf402e040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efde27fc640 1 -- 172.21.15.73:0/3318395910 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf001c480 con 0x5623eb4b6080 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efde27fc640 1 -- 172.21.15.73:0/3318395910 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf001cb00 con 0x5623eb4b6080 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.151+0000 7efde27fc640 1 -- 172.21.15.73:0/3318395910 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0011df0 con 0x5623eb4b6080 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- 172.21.15.73:0/3318395910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 msgr2=0x5623eb4be720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 crypto rx=0x7efdf000bcb0 tx=0x7efdf00107a0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- 172.21.15.73:0/3318395910 shutdown_connections 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4bec60 0x5623eb4c1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b6080 0x5623eb4be720 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 --2- 172.21.15.73:0/3318395910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4b5b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- 172.21.15.73:0/3318395910 >> 172.21.15.73:0/3318395910 conn(0x5623eb4afe90 msgr2=0x5623eb4b0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- 172.21.15.73:0/3318395910 shutdown_connections 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- 172.21.15.73:0/3318395910 wait complete. 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.152+0000 7efe055b7740 1 -- start start 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 Processor -- start 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bb700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4bbc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb4c1a50 con 0x5623eb4b5760 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb3d50a0 con 0x5623eb4bec60 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5623eb4c1bd0 con 0x5623eb4b6080 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efe055b7740 10 client.0 _ll_register_callbacks cb 0x5623eb3d4a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56918/0 (socket says 172.21.15.73:56918) 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 -- 172.21.15.73:0/1506219065 learned_addr learned my addr 172.21.15.73:0/1506219065 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bb700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 -- 172.21.15.73:0/1506219065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 msgr2=0x5623eb4bb700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bb700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 -- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 msgr2=0x5623eb4bbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4bbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfbfff640 1 -- 172.21.15.73:0/1506219065 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623eb12d3f0 con 0x5623eb4b5760 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.153+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:18.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efdfbfff640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efdf00116b0 tx=0x7efdf000bfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=243d80427269c332 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf001c480 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf001cb00 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efdf001c620 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf0032050 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 10 client.15921 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5623eb0ed0f0 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf001cb00 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efdf002d030 con 0x5623eb4b5760 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 10 client.15921 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 10 client.15921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 20 client.15921 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb4ce650 0x5623eb4d0a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 4 client.15921 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.154+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5623eb426340 con 0x5623eb4ce650 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.155+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb4ce650 0x5623eb4d0a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.155+0000 7efddb7fe640 10 client.15921 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.155+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb4ce650 0x5623eb4d0a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=184426fbe3a69363 server_cookie=f37b055fd55379a3 in_seq=0 out_seq=0 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efddb7fe640 1 -- 172.21.15.73:0/1506219065 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5623eb426340 con 0x5623eb4ce650 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efddb7fe640 10 client.15921 handle_command_reply: tid=0 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 client.15921 shutdown 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 msgr2=0x5623eb4be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 crypto rx=0x7efdf00116b0 tx=0x7efdf000bfd0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 shutdown_connections 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb4ce650 0x5623eb4d0a40 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4bbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bb700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 --2- 172.21.15.73:0/1506219065 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4be370 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 >> 172.21.15.73:0/1506219065 conn(0x5623eb4afe90 msgr2=0x5623eb4b1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efddb7fe640 0 client.15921 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 shutdown_connections 2024-03-09T00:08:18.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 1 -- 172.21.15.73:0/1506219065 wait complete. 2024-03-09T00:08:18.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.156+0000 7efe055b7740 20 client.15921 trim_cache size 0 max 16384 2024-03-09T00:08:18.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 Processor -- start 2024-03-09T00:08:18.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 -- start start 2024-03-09T00:08:18.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4ba6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bde30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1ac6b0 con 0x5623eb4bec60 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1aec70 con 0x5623eb4b5760 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe055b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1accb0 con 0x5623eb4b6080 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe00ea2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe00ea2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56930/0 (socket says 172.21.15.73:56930) 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.171+0000 7efe00ea2640 1 -- 172.21.15.73:0/3471169892 learned_addr learned my addr 172.21.15.73:0/3471169892 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bde30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 -- 172.21.15.73:0/3471169892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 msgr2=0x5623eb4bde30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bde30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efdfbfff640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4ba6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 -- 172.21.15.73:0/3471169892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 msgr2=0x5623eb4ba6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4ba6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 -- 172.21.15.73:0/3471169892 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5623eb4ab760 con 0x5623eb4bec60 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efdfbfff640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4ba6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efe00ea2640 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7efdf4004af0 tx=0x7efdf40095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4502ffc2fbc004c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efde1ffb640 1 -- 172.21.15.73:0/3471169892 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4039040 con 0x5623eb4bec60 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efde1ffb640 1 -- 172.21.15.73:0/3471169892 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf4052070 con 0x5623eb4bec60 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.172+0000 7efde1ffb640 1 -- 172.21.15.73:0/3471169892 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf4004850 con 0x5623eb4bec60 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- 172.21.15.73:0/3471169892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 msgr2=0x5623eb4be370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 crypto rx=0x7efdf4004af0 tx=0x7efdf40095c0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- 172.21.15.73:0/3471169892 shutdown_connections 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4bec60 0x5623eb4be370 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4bde30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 --2- 172.21.15.73:0/3471169892 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4b5760 0x5623eb4ba6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- 172.21.15.73:0/3471169892 >> 172.21.15.73:0/3471169892 conn(0x5623eb4afe90 msgr2=0x5623eb4b9fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- 172.21.15.73:0/3471169892 shutdown_connections 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- 172.21.15.73:0/3471169892 wait complete. 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 -- start start 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.173+0000 7efe055b7740 1 Processor -- start 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4c2c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4c31c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1ad2f0 con 0x5623eb4b5760 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1ae610 con 0x5623eb4bec60 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdfc1ad950 con 0x5623eb4b6080 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 10 client.0 _ll_register_callbacks cb 0x7efdf000bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:18.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56934/0 (socket says 172.21.15.73:56934) 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 -- 172.21.15.73:0/753414756 learned_addr learned my addr 172.21.15.73:0/753414756 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4c2c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe00ea2640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4c31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 -- 172.21.15.73:0/753414756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 msgr2=0x5623eb4c2c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4c2c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 -- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 msgr2=0x5623eb4c31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4c31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 -- 172.21.15.73:0/753414756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5623eb524840 con 0x5623eb4b5760 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efdfbfff640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005d00 tx=0x7efdf0011590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=daca149726f6eb35 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efddb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efdf0010cd0 con 0x5623eb4b5760 2024-03-09T00:08:18.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7efdf0004020 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.174+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdf4052070 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdf0044050 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 10 client.15927 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5623eb525a00 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efdf4052070 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7efdf002d040 con 0x5623eb4b5760 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 10 client.15927 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 10 client.15927 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 20 client.15927 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb526060 0x5623eb528480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 4 client.15927 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5623eb420690 con 0x5623eb526060 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.175+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb526060 0x5623eb528480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efddb7fe640 10 client.15927 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efdfb7fe640 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb526060 0x5623eb528480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fccc2173082359af server_cookie=6e6febbfb1fedd61 in_seq=0 out_seq=0 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efddb7fe640 1 -- 172.21.15.73:0/753414756 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5623eb420690 con 0x5623eb526060 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efddb7fe640 10 client.15927 handle_command_reply: tid=0 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 client.15927 shutdown 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 msgr2=0x5623eb4c2740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 secure :-1 s=READY pgs=600 cs=0 l=1 rev1=1 crypto rx=0x7efdf0005d00 tx=0x7efdf0011590 comp rx=0 tx=0).stop 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 shutdown_connections 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5623eb526060 0x5623eb528480 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5623eb4bec60 0x5623eb4c31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efddb7fe640 0 client.15927 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5623eb4b6080 0x5623eb4c2c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 --2- 172.21.15.73:0/753414756 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5623eb4b5760 0x5623eb4c2740 unknown :-1 s=CLOSED pgs=600 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 >> 172.21.15.73:0/753414756 conn(0x5623eb4afe90 msgr2=0x5623eb4ba860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 shutdown_connections 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 1 -- 172.21.15.73:0/753414756 wait complete. 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.176+0000 7efe055b7740 20 client.15927 trim_cache size 0 max 16384 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:08:18.178 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 msgr2=0x7efde40a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 0x7efde40a4f50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7efdf4005cc0 tx=0x7efdf402e040 comp rx=0 tx=0).stop 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 msgr2=0x7efdfc19dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 crypto rx=0x7efdec00e9f0 tx=0x7efdec00edf0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 shutdown_connections 2024-03-09T00:08:18.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7efdfc1074f0 0x7efdfc197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7efde40a2a90 0x7efde40a4f50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7efdfc100570 0x7efdfc19dd20 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 --2- 172.21.15.73:0/4211056040 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7efdfc0ffbb0 0x7efdfc19d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 >> 172.21.15.73:0/4211056040 conn(0x7efdfc075530 msgr2=0x7efdfc0fe5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 shutdown_connections 2024-03-09T00:08:18.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.177+0000 7efe02905640 1 -- 172.21.15.73:0/4211056040 wait complete. 2024-03-09T00:08:18.183 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:08:18.183 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 -- 172.21.15.73:0/2200376107 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100db0 msgr2=0x7f6ddc104df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 --2- 172.21.15.73:0/2200376107 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100db0 0x7f6ddc104df0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8009900 tx=0x7f6dd802bc30 comp rx=0 tx=0).stop 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 -- 172.21.15.73:0/2200376107 shutdown_connections 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 --2- 172.21.15.73:0/2200376107 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1054d0 0x7f6ddc10d0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 --2- 172.21.15.73:0/2200376107 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100db0 0x7f6ddc104df0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 --2- 172.21.15.73:0/2200376107 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc100400 0x7f6ddc1007e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 -- 172.21.15.73:0/2200376107 >> 172.21.15.73:0/2200376107 conn(0x7f6ddc076570 msgr2=0x7f6ddc076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 -- 172.21.15.73:0/2200376107 shutdown_connections 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 -- 172.21.15.73:0/2200376107 wait complete. 2024-03-09T00:08:18.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.292+0000 7f6de479e640 1 Processor -- start 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- start start 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc1054d0 0x7f6ddc19cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1968a0 0x7f6ddc196d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc110560 con 0x7f6ddc100400 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1103e0 con 0x7f6ddc1054d0 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ddc1106e0 con 0x7f6ddc1968a0 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56946/0 (socket says 172.21.15.73:56946) 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 -- 172.21.15.73:0/3653972197 learned_addr learned my addr 172.21.15.73:0/3653972197 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de1d39640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc1054d0 0x7f6ddc19cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de2d3b640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1968a0 0x7f6ddc196d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 -- 172.21.15.73:0/3653972197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1968a0 msgr2=0x7f6ddc196d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1968a0 0x7f6ddc196d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 -- 172.21.15.73:0/3653972197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc1054d0 msgr2=0x7f6ddc19cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc1054d0 0x7f6ddc19cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 -- 172.21.15.73:0/3653972197 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ddc1973c0 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de253a640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00d950 tx=0x7f6dcc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc017070 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6dcc009c30 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ddc1976b0 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6dcc00f6b0 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.293+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ddc1a3900 con 0x7f6ddc100400 2024-03-09T00:08:18.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.294+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ddc1018d0 con 0x7f6ddc100400 2024-03-09T00:08:18.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.296+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f6dcc0109d0 con 0x7f6ddc100400 2024-03-09T00:08:18.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.297+0000 7f6dd37fe640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 0x7f6dc80a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.297+0000 7f6de1d39640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 0x7f6dc80a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.297+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6dcc0cf180 con 0x7f6ddc100400 2024-03-09T00:08:18.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.297+0000 7f6de1d39640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 0x7f6dc80a4f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8006010 tx=0x7f6dd8000bc0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.300+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f6dcc009de0 con 0x7f6ddc100400 2024-03-09T00:08:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:18 smithi179 ceph-mon[36637]: pgmap v46: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-03-09T00:08:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:18 smithi073 ceph-mon[32309]: pgmap v46: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-03-09T00:08:18.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.512+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6ddc068530 con 0x7f6ddc100400 2024-03-09T00:08:18.507 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.512+0000 7f6dd37fe640 1 -- 172.21.15.73:0/3653972197 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f6dcc09b900 con 0x7f6ddc100400 2024-03-09T00:08:18.507 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:08:18.507 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:08:18.508 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:08:18.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:18 smithi102 ceph-mon[36106]: pgmap v46: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 23 op/s 2024-03-09T00:08:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 msgr2=0x7f6dc80a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 0x7f6dc80a4f50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f6dd8006010 tx=0x7f6dd8000bc0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 msgr2=0x7f6ddc19c6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 crypto rx=0x7f6dcc00d950 tx=0x7f6dcc00de10 comp rx=0 tx=0).stop 2024-03-09T00:08:18.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 shutdown_connections 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ddc1968a0 0x7f6ddc196d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ddc1054d0 0x7f6ddc19cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6dc80a2a90 0x7f6dc80a4f50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 --2- 172.21.15.73:0/3653972197 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ddc100400 0x7f6ddc19c6f0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.514+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 >> 172.21.15.73:0/3653972197 conn(0x7f6ddc076570 msgr2=0x7f6ddc0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.515+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 shutdown_connections 2024-03-09T00:08:18.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.515+0000 7f6de479e640 1 -- 172.21.15.73:0/3653972197 wait complete. 2024-03-09T00:08:18.520 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:08:18.521 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:08:18.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- 172.21.15.73:0/49241834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901065a0 msgr2=0x7f619010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 --2- 172.21.15.73:0/49241834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901065a0 0x7f619010e170 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 crypto rx=0x7f6188009ca0 tx=0x7f618802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- 172.21.15.73:0/49241834 shutdown_connections 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 --2- 172.21.15.73:0/49241834 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901065a0 0x7f619010e170 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 --2- 172.21.15.73:0/49241834 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6190101e80 0x7f6190105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 --2- 172.21.15.73:0/49241834 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901014d0 0x7f61901018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- 172.21.15.73:0/49241834 >> 172.21.15.73:0/49241834 conn(0x7f61900fcd90 msgr2=0x7f61900ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- 172.21.15.73:0/49241834 shutdown_connections 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- 172.21.15.73:0/49241834 wait complete. 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 Processor -- start 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 -- start start 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.630+0000 7f6194937640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901065a0 0x7f619019dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61901979d0 0x7f6190197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901116b0 con 0x7f61901014d0 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190111530 con 0x7f61901065a0 2024-03-09T00:08:18.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6190111830 con 0x7f61901979d0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42964/0 (socket says 172.21.15.73:42964) 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 -- 172.21.15.73:0/2768087362 learned_addr learned my addr 172.21.15.73:0/2768087362 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6187fff640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901065a0 0x7f619019dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618ed9d640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61901979d0 0x7f6190197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 -- 172.21.15.73:0/2768087362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61901979d0 msgr2=0x7f6190197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61901979d0 0x7f6190197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 -- 172.21.15.73:0/2768087362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901065a0 msgr2=0x7f619019dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901065a0 0x7f619019dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 -- 172.21.15.73:0/2768087362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61901984f0 con 0x7f61901014d0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f618e59c640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f618000d950 tx=0x7f618000dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f61877fe640 1 -- 172.21.15.73:0/2768087362 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180017070 con 0x7f61901014d0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f61877fe640 1 -- 172.21.15.73:0/2768087362 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f618000de80 con 0x7f61901014d0 2024-03-09T00:08:18.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61901987e0 con 0x7f61901014d0 2024-03-09T00:08:18.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f61877fe640 1 -- 172.21.15.73:0/2768087362 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f618000f620 con 0x7f61901014d0 2024-03-09T00:08:18.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.631+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61901a49e0 con 0x7f61901014d0 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.633+0000 7f61877fe640 1 -- 172.21.15.73:0/2768087362 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7f6180009c30 con 0x7f61901014d0 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 Processor -- start 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 -- start start 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61877fe640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 0x7f617c0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x559062534160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x559062536a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590624f2360 con 0x55906252b1a0 2024-03-09T00:08:18.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55906248eec0 con 0x55906252bac0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61975ef740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559062437d30 con 0x5590625346a0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618ed9d640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 0x7f617c0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42970/0 (socket says 172.21.15.73:42970) 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618e59c640 1 -- 172.21.15.73:0/1088747415 learned_addr learned my addr 172.21.15.73:0/1088747415 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f61877fe640 1 -- 172.21.15.73:0/2768087362 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f61800cf160 con 0x7f61901014d0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f6187fff640 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x559062534160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618ed9d640 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x559062536a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618e59c640 1 -- 172.21.15.73:0/1088747415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 msgr2=0x559062536a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.634+0000 7f618e59c640 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x559062536a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f618e59c640 1 -- 172.21.15.73:0/1088747415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 msgr2=0x559062534160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f618e59c640 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x559062534160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f618e59c640 1 -- 172.21.15.73:0/1088747415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559062436bc0 con 0x55906252b1a0 2024-03-09T00:08:18.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f618ed9d640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 0x7f617c0a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6188005cc0 tx=0x7f618802cee0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f618e59c640 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f618009aee0 tx=0x7f61800509e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=184e99a5bf58b692 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f6184ff9640 1 -- 172.21.15.73:0/1088747415 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180020030 con 0x55906252b1a0 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f6184ff9640 1 -- 172.21.15.73:0/1088747415 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f618009b1e0 con 0x55906252b1a0 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.635+0000 7f6184ff9640 1 -- 172.21.15.73:0/1088747415 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800cc680 con 0x55906252b1a0 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- 172.21.15.73:0/1088747415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 msgr2=0x55906252b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 crypto rx=0x7f618009aee0 tx=0x7f61800509e0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- 172.21.15.73:0/1088747415 shutdown_connections 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x559062536a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x559062534160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- 172.21.15.73:0/1088747415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x55906252b580 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- 172.21.15.73:0/1088747415 >> 172.21.15.73:0/1088747415 conn(0x5590625258d0 msgr2=0x559062525cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- 172.21.15.73:0/1088747415 shutdown_connections 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- 172.21.15.73:0/1088747415 wait complete. 2024-03-09T00:08:18.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- start start 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 Processor -- start 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252bac0 0x559062531cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55906242d4a0 con 0x5590625346a0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55906248af60 con 0x55906252b1a0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559062537430 con 0x55906252bac0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f61975ef740 10 client.0 _ll_register_callbacks cb 0x559062488d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f618e59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f618ed9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f618e59c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:43530/0 (socket says 172.21.15.73:43530) 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f618ed9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42972/0 (socket says 172.21.15.73:42972) 2024-03-09T00:08:18.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f618e59c640 1 -- 172.21.15.73:0/714225378 learned_addr learned my addr 172.21.15.73:0/714225378 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.636+0000 7f6187fff640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252bac0 0x559062531cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 -- 172.21.15.73:0/714225378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252bac0 msgr2=0x559062531cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252bac0 0x559062531cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 -- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 msgr2=0x559062531770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 -- 172.21.15.73:0/714225378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590621a33f0 con 0x5590625346a0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618e59c640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f618ed9d640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f618802c700 tx=0x7f618802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=41ac296d94f84c1d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188052060 con 0x5590625346a0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6188004240 con 0x5590625346a0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f618009b1e0 con 0x5590625346a0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6188059050 con 0x5590625346a0 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f61975ef740 10 client.15945 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:18.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55906226d7d0 con 0x5590625346a0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f618009b1e0 con 0x5590625346a0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f618804d920 con 0x5590625346a0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.637+0000 7f61975ef740 10 client.15945 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f61975ef740 10 client.15945 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f61975ef740 20 client.15945 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5590625448d0 0x559062546d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f61975ef740 4 client.15945 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55906249c400 con 0x5590625448d0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f618e59c640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5590625448d0 0x559062546d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f618e59c640 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5590625448d0 0x559062546d10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=12d41b72fb5c1d35 server_cookie=6a659965bedefd38 in_seq=0 out_seq=0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.638+0000 7f6175ffb640 10 client.15945 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f6175ffb640 1 -- 172.21.15.73:0/714225378 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55906249c400 con 0x5590625448d0 2024-03-09T00:08:18.640 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f6175ffb640 10 client.15945 handle_command_reply: tid=0 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 client.15945 shutdown 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 msgr2=0x5590625321f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f618802c700 tx=0x7f618802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 shutdown_connections 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5590625448d0 0x559062546d10 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x5590625321f0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f6175ffb640 0 client.15945 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252bac0 0x559062531cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 --2- 172.21.15.73:0/714225378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252b1a0 0x559062531770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 >> 172.21.15.73:0/714225378 conn(0x5590625258d0 msgr2=0x559062527620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 shutdown_connections 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 1 -- 172.21.15.73:0/714225378 wait complete. 2024-03-09T00:08:18.641 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.639+0000 7f61975ef740 20 client.15945 trim_cache size 0 max 16384 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.650+0000 7f61975ef740 1 Processor -- start 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.650+0000 7f61975ef740 1 -- start start 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.650+0000 7f61975ef740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f61975ef740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906252e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f61975ef740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x55906252ef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f61975ef740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901ac850 con 0x55906252b1a0 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f61975ef740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901aee10 con 0x55906252bac0 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f61975ef740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901ace50 con 0x5590625346a0 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42984/0 (socket says 172.21.15.73:42984) 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 -- 172.21.15.73:0/1185198057 learned_addr learned my addr 172.21.15.73:0/1185198057 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618ed9d640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x55906252ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 -- 172.21.15.73:0/1185198057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 msgr2=0x55906252ef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x55906252ef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f6187fff640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906252e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 -- 172.21.15.73:0/1185198057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 msgr2=0x55906252e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906252e9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 -- 172.21.15.73:0/1185198057 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559062536e00 con 0x55906252b1a0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f6187fff640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906252e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f618e59c640 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f618009aee0 tx=0x7f6180000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b83ff7b5fe459aa server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f6177fff640 1 -- 172.21.15.73:0/1185198057 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6180020040 con 0x55906252b1a0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f6177fff640 1 -- 172.21.15.73:0/1185198057 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f618009b1e0 con 0x55906252b1a0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.651+0000 7f6177fff640 1 -- 172.21.15.73:0/1185198057 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61800cc680 con 0x55906252b1a0 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- 172.21.15.73:0/1185198057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 msgr2=0x559062533ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 crypto rx=0x7f618009aee0 tx=0x7f6180000bc0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- 172.21.15.73:0/1185198057 shutdown_connections 2024-03-09T00:08:18.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5590625346a0 0x55906252ef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906252e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- 172.21.15.73:0/1185198057 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55906252b1a0 0x559062533ea0 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- 172.21.15.73:0/1185198057 >> 172.21.15.73:0/1185198057 conn(0x5590625258d0 msgr2=0x55906254b670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- 172.21.15.73:0/1185198057 shutdown_connections 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- 172.21.15.73:0/1185198057 wait complete. 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- start start 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 Processor -- start 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252b1a0 0x55906253a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906253a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901ad490 con 0x5590625346a0 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901ae7b0 con 0x55906252bac0 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61901adaf0 con 0x55906252b1a0 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f61975ef740 10 client.0 _ll_register_callbacks cb 0x7f6180010c10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:18.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f618ed9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f618ed9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42994/0 (socket says 172.21.15.73:42994) 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f618ed9d640 1 -- 172.21.15.73:0/457847192 learned_addr learned my addr 172.21.15.73:0/457847192 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f618e59c640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252b1a0 0x55906253a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.652+0000 7f6187fff640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906253a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 -- 172.21.15.73:0/457847192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252b1a0 msgr2=0x55906253a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252b1a0 0x55906253a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 -- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 msgr2=0x55906253a880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906253a880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 -- 172.21.15.73:0/457847192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55906253b130 con 0x5590625346a0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f618ed9d640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6188005c70 tx=0x7f61880093f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6898b71761d7a6d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6188052070 con 0x5590625346a0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f618009b1e0 con 0x5590625346a0 2024-03-09T00:08:18.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f618804d850 con 0x5590625346a0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6188039040 con 0x5590625346a0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 10 client.15951 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55906259c390 con 0x5590625346a0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f618009b1e0 con 0x5590625346a0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f618805e4d0 con 0x5590625346a0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 10 client.15951 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 10 client.15951 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 20 client.15951 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55906259ca20 0x55906259ee40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 4 client.15951 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.653+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559062496750 con 0x55906259ca20 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f618e59c640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55906259ca20 0x55906259ee40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f6175ffb640 10 client.15951 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f618e59c640 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55906259ca20 0x55906259ee40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=761cae9e51a756b7 server_cookie=ddd88bb6f01a7bf2 in_seq=0 out_seq=0 2024-03-09T00:08:18.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f6175ffb640 1 -- 172.21.15.73:0/457847192 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559062496750 con 0x55906259ca20 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f6175ffb640 10 client.15951 handle_command_reply: tid=0 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 client.15951 shutdown 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 msgr2=0x55906259ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 crypto rx=0x7f6188005c70 tx=0x7f61880093f0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 shutdown_connections 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55906259ca20 0x55906259ee40 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5590625346a0 0x55906259ae00 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f6175ffb640 0 client.15951 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55906252bac0 0x55906253a880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 --2- 172.21.15.73:0/457847192 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55906252b1a0 0x55906253a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 >> 172.21.15.73:0/457847192 conn(0x5590625258d0 msgr2=0x559062531730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 shutdown_connections 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 1 -- 172.21.15.73:0/457847192 wait complete. 2024-03-09T00:08:18.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.654+0000 7f61975ef740 20 client.15951 trim_cache size 0 max 16384 2024-03-09T00:08:18.657 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 msgr2=0x7f617c0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 0x7f617c0a4f50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f6188005cc0 tx=0x7f618802cee0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 msgr2=0x7f619019d7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 crypto rx=0x7f618000d950 tx=0x7f618000dd50 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 shutdown_connections 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f61901979d0 0x7f6190197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f61901065a0 0x7f619019dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f617c0a2a90 0x7f617c0a4f50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 --2- 172.21.15.73:0/2768087362 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f61901014d0 0x7f619019d7d0 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 >> 172.21.15.73:0/2768087362 conn(0x7f61900fcd90 msgr2=0x7f61900ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 shutdown_connections 2024-03-09T00:08:18.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:18.655+0000 7f6194937640 1 -- 172.21.15.73:0/2768087362 wait complete. 2024-03-09T00:08:19.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3653972197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:19.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3653972197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:19.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3653972197' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:19.661 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:08:19.661 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:08:19.661 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:08:19.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 -- 172.21.15.73:0/3338283953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2141004a0 msgr2=0x7ff21410c680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 --2- 172.21.15.73:0/3338283953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2141004a0 0x7ff21410c680 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 crypto rx=0x7ff20c00adb0 tx=0x7ff20c02d090 comp rx=0 tx=0).stop 2024-03-09T00:08:19.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 -- 172.21.15.73:0/3338283953 shutdown_connections 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 --2- 172.21.15.73:0/3338283953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2141004a0 0x7ff21410c680 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 --2- 172.21.15.73:0/3338283953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2140ffae0 0x7ff2140fff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 --2- 172.21.15.73:0/3338283953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 0x7ff214105ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 -- 172.21.15.73:0/3338283953 >> 172.21.15.73:0/3338283953 conn(0x7ff2140753d0 msgr2=0x7ff2140757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 -- 172.21.15.73:0/3338283953 shutdown_connections 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.774+0000 7ff2191a2640 1 -- 172.21.15.73:0/3338283953 wait complete. 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 Processor -- start 2024-03-09T00:08:19.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 -- start start 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2141004a0 0x7ff21419db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 0x7ff2141977e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410f370 con 0x7ff2140ffae0 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410f1f0 con 0x7ff2141004a0 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2191a2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff21410f4f0 con 0x7ff2141058c0 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43022/0 (socket says 172.21.15.73:43022) 2024-03-09T00:08:19.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff213fff640 1 -- 172.21.15.73:0/1023804564 learned_addr learned my addr 172.21.15.73:0/1023804564 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2137fe640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2141004a0 0x7ff21419db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.775+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 0x7ff2141977e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 -- 172.21.15.73:0/1023804564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 msgr2=0x7ff2141977e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 0x7ff2141977e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 -- 172.21.15.73:0/1023804564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2141004a0 msgr2=0x7ff21419db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2141004a0 0x7ff21419db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 -- 172.21.15.73:0/1023804564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff214197fb0 con 0x7ff2140ffae0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff213fff640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff20400d770 tx=0x7ff20400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff2117fa640 1 -- 172.21.15.73:0/1023804564 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204017070 con 0x7ff2140ffae0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2141982a0 con 0x7ff2140ffae0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff2117fa640 1 -- 172.21.15.73:0/1023804564 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff204005500 con 0x7ff2140ffae0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff2117fa640 1 -- 172.21.15.73:0/1023804564 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204004e00 con 0x7ff2140ffae0 2024-03-09T00:08:19.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.776+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2141a4470 con 0x7ff2140ffae0 2024-03-09T00:08:19.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.778+0000 7ff2117fa640 1 -- 172.21.15.73:0/1023804564 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7ff204004fa0 con 0x7ff2140ffae0 2024-03-09T00:08:19.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.778+0000 7ff21cfe6740 1 Processor -- start 2024-03-09T00:08:19.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.778+0000 7ff21cfe6740 1 -- start start 2024-03-09T00:08:19.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.778+0000 7ff2117fa640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 0x7ff1fc0a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.778+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfd390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c05f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2b60b40 con 0x5628b2c064b0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2b09d80 con 0x5628b2bfd8d0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2b5cbe0 con 0x5628b2bfcfb0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 0x7ff1fc0a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43030/0 (socket says 172.21.15.73:43030) 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 -- 172.21.15.73:0/1338328202 learned_addr learned my addr 172.21.15.73:0/1338328202 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2137fe640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c05f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2117fa640 1 -- 172.21.15.73:0/1023804564 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff2040cef00 con 0x7ff2140ffae0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff213fff640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfd390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 -- 172.21.15.73:0/1338328202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfcfb0 msgr2=0x5628b2bfd390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfd390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 -- 172.21.15.73:0/1338328202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 msgr2=0x5628b2c05f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c05f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 -- 172.21.15.73:0/1338328202 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b293d7e0 con 0x5628b2c064b0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2137fe640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c05f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 0x7ff1fc0a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff20c038040 tx=0x7ff20c00a990 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff2189a1640 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff20c037000 tx=0x7ff20c046000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96142b81b41a7f11 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff1fa7fc640 1 -- 172.21.15.73:0/1338328202 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c051070 con 0x5628b2c064b0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff1fa7fc640 1 -- 172.21.15.73:0/1338328202 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c03b040 con 0x5628b2c064b0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.779+0000 7ff1fa7fc640 1 -- 172.21.15.73:0/1338328202 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20c03a070 con 0x5628b2c064b0 2024-03-09T00:08:19.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- 172.21.15.73:0/1338328202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 msgr2=0x5628b2c088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 crypto rx=0x7ff20c037000 tx=0x7ff20c046000 comp rx=0 tx=0).stop 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- 172.21.15.73:0/1338328202 shutdown_connections 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2c064b0 0x5628b2c088a0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c05f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/1338328202 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfd390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- 172.21.15.73:0/1338328202 >> 172.21.15.73:0/1338328202 conn(0x5628b2bf76e0 msgr2=0x5628b2bf7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- 172.21.15.73:0/1338328202 shutdown_connections 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- 172.21.15.73:0/1338328202 wait complete. 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.780+0000 7ff21cfe6740 1 -- start start 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 Processor -- start 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c03230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2c03840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2c09240 con 0x5628b2bfcfb0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2b1d1c0 con 0x5628b2bfd8d0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628b2c093c0 con 0x5628b2c064b0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff21cfe6740 10 client.0 _ll_register_callbacks cb 0x5628b2b0c7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43032/0 (socket says 172.21.15.73:43032) 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 -- 172.21.15.73:0/2116715063 learned_addr learned my addr 172.21.15.73:0/2116715063 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff2189a1640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2c03840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff2137fe640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c03230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 -- 172.21.15.73:0/2116715063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 msgr2=0x5628b2c03840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2c03840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 -- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 msgr2=0x5628b2c03230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c03230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 -- 172.21.15.73:0/2116715063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628b299df70 con 0x5628b2bfcfb0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff213fff640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff20409af10 tx=0x7ff204097330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=151a734fa2a75ac7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.781+0000 7ff1f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204020040 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff2040975f0 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff20c03b040 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff2040b4220 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 10 client.15963 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5628b2835170 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff20c03b040 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff2040b4400 con 0x5628b2bfcfb0 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 10 client.15963 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 10 client.15963 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 20 client.15963 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c16550 0x5628b2c18960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 4 client.15963 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:19.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628b2b36960 con 0x5628b2c16550 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.782+0000 7ff2137fe640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c16550 0x5628b2c18960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff1f37fe640 10 client.15963 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff2137fe640 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c16550 0x5628b2c18960 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=838b763fe3d4ff54 server_cookie=5575825255a9419b in_seq=0 out_seq=0 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff1f37fe640 1 -- 172.21.15.73:0/2116715063 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5628b2b36960 con 0x5628b2c16550 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff1f37fe640 10 client.15963 handle_command_reply: tid=0 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 client.15963 shutdown 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 msgr2=0x5628b2c05ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 crypto rx=0x7ff20409af10 tx=0x7ff204097330 comp rx=0 tx=0).stop 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 shutdown_connections 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c16550 0x5628b2c18960 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2c03840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfd8d0 0x5628b2c03230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/2116715063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c05ea0 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff1f37fe640 0 client.15963 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 >> 172.21.15.73:0/2116715063 conn(0x5628b2bf76e0 msgr2=0x5628b2bf9430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 shutdown_connections 2024-03-09T00:08:19.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.783+0000 7ff21cfe6740 1 -- 172.21.15.73:0/2116715063 wait complete. 2024-03-09T00:08:19.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.784+0000 7ff21cfe6740 20 client.15963 trim_cache size 0 max 16384 2024-03-09T00:08:19.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 Processor -- start 2024-03-09T00:08:19.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 -- start start 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfdcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2bfea50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141ac530 con 0x5628b2bfd8d0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141aeaf0 con 0x5628b2bfcfb0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141acb30 con 0x5628b2c064b0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.798+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43036/0 (socket says 172.21.15.73:43036) 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 -- 172.21.15.73:0/3472785646 learned_addr learned my addr 172.21.15.73:0/3472785646 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2189a1640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2bfea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff213fff640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfdcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 -- 172.21.15.73:0/3472785646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 msgr2=0x5628b2bfea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2bfea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 -- 172.21.15.73:0/3472785646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 msgr2=0x5628b2bfdcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfdcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 -- 172.21.15.73:0/3472785646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628b2bfd700 con 0x5628b2bfd8d0 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2189a1640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2bfea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff213fff640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff208000c00 tx=0x7ff20800b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10c3c6b58746a3d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff1f9ffb640 1 -- 172.21.15.73:0/3472785646 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff208003ca0 con 0x5628b2bfd8d0 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff1f9ffb640 1 -- 172.21.15.73:0/3472785646 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff208023030 con 0x5628b2bfd8d0 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.799+0000 7ff1f9ffb640 1 -- 172.21.15.73:0/3472785646 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff20801f0f0 con 0x5628b2bfd8d0 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3472785646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 msgr2=0x5628b2bfe510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 crypto rx=0x7ff208000c00 tx=0x7ff20800b390 comp rx=0 tx=0).stop 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3472785646 shutdown_connections 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2c064b0 0x5628b2bfea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfd8d0 0x5628b2bfe510 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3472785646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2bfcfb0 0x5628b2bfdcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3472785646 >> 172.21.15.73:0/3472785646 conn(0x5628b2bf76e0 msgr2=0x5628b2bf7ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3472785646 shutdown_connections 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3472785646 wait complete. 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- start start 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 Processor -- start 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 0x5628b2c0d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2c064b0 0x5628b2c0a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141ad170 con 0x5628b2bfcfb0 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141ae490 con 0x5628b2c064b0 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2141ad7d0 con 0x5628b2bfd8d0 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.800+0000 7ff21cfe6740 10 client.0 _ll_register_callbacks cb 0x7ff20c038070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43042/0 (socket says 172.21.15.73:43042) 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 -- 172.21.15.73:0/3636335444 learned_addr learned my addr 172.21.15.73:0/3636335444 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff2189a1640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2c064b0 0x5628b2c0a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 0x5628b2c0d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 -- 172.21.15.73:0/3636335444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 msgr2=0x5628b2c0d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 0x5628b2c0d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 -- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2c064b0 msgr2=0x5628b2c0a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2c064b0 0x5628b2c0a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 -- 172.21.15.73:0/3636335444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628b2c0aee0 con 0x5628b2bfcfb0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 0x5628b2c0d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff213fff640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff20409ac80 tx=0x7ff204099490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b32fc1cc3bfc6f9b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff1f37fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff204020030 con 0x5628b2bfcfb0 2024-03-09T00:08:19.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff2040cc420 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff208023030 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff204083050 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff21cfe6740 10 client.15969 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5628b2c6e510 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.801+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff208023030 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff20409a3d0 con 0x5628b2bfcfb0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 10 client.15969 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 10 client.15969 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 20 client.15969 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c6eb00 0x5628b2c70f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 4 client.15969 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5628b2b22080 con 0x5628b2c6eb00 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c6eb00 0x5628b2c70f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff1f37fe640 10 client.15969 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.802+0000 7ff2137fe640 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c6eb00 0x5628b2c70f20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=983a9bd37eb0ad54 server_cookie=7f2682c3f60de353 in_seq=0 out_seq=0 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff1f37fe640 1 -- 172.21.15.73:0/3636335444 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5628b2b22080 con 0x5628b2c6eb00 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff1f37fe640 10 client.15969 handle_command_reply: tid=0 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 client.15969 shutdown 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 msgr2=0x5628b2c0d2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 crypto rx=0x7ff20409ac80 tx=0x7ff204099490 comp rx=0 tx=0).stop 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 shutdown_connections 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5628b2c6eb00 0x5628b2c70f20 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5628b2c064b0 0x5628b2c0a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5628b2bfd8d0 0x5628b2c0d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff1f37fe640 0 client.15969 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 --2- 172.21.15.73:0/3636335444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5628b2bfcfb0 0x5628b2c0d2d0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 >> 172.21.15.73:0/3636335444 conn(0x5628b2bf76e0 msgr2=0x5628b2c016d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 shutdown_connections 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 1 -- 172.21.15.73:0/3636335444 wait complete. 2024-03-09T00:08:19.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.803+0000 7ff21cfe6740 20 client.15969 trim_cache size 0 max 16384 2024-03-09T00:08:19.805 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:19.805 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:08:19.805 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "2e51ff4d-21ea-4e42-bd20-1bf8f00dcc8c", 2024-03-09T00:08:19.805 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:08:19.805 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 msgr2=0x7ff1fc0a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 0x7ff1fc0a4f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7ff20c038040 tx=0x7ff20c00a990 comp rx=0 tx=0).stop 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 msgr2=0x7ff21419d620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 crypto rx=0x7ff20400d770 tx=0x7ff20400dc30 comp rx=0 tx=0).stop 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 shutdown_connections 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff2141058c0 0x7ff2141977e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff2141004a0 0x7ff21419db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff1fc0a2a90 0x7ff1fc0a4f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 --2- 172.21.15.73:0/1023804564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff2140ffae0 0x7ff21419d620 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 >> 172.21.15.73:0/1023804564 conn(0x7ff2140753d0 msgr2=0x7ff214103b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 shutdown_connections 2024-03-09T00:08:19.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.804+0000 7ff2191a2640 1 -- 172.21.15.73:0/1023804564 wait complete. 2024-03-09T00:08:19.809 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:08:19.809 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 -- 172.21.15.73:0/2044493280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088100810 msgr2=0x7fc0881045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/2044493280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088100810 0x7fc0881045b0 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009900 tx=0x7fc07c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 -- 172.21.15.73:0/2044493280 shutdown_connections 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/2044493280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc088104ce0 0x7fc08810c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/2044493280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088100810 0x7fc0881045b0 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/2044493280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0880ffe60 0x7fc088100240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 -- 172.21.15.73:0/2044493280 >> 172.21.15.73:0/2044493280 conn(0x7fc088075450 msgr2=0x7fc088075860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 -- 172.21.15.73:0/2044493280 shutdown_connections 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.923+0000 7fc08d9b3640 1 -- 172.21.15.73:0/2044493280 wait complete. 2024-03-09T00:08:19.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 Processor -- start 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 -- start start 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0880ffe60 0x7fc08819d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc088104ce0 0x7fc08819ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881116c0 con 0x7fc088197a20 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc088111540 con 0x7fc088104ce0 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc08d9b3640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc088111840 con 0x7fc0880ffe60 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43064/0 (socket says 172.21.15.73:43064) 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc0877fe640 1 -- 172.21.15.73:0/3214227118 learned_addr learned my addr 172.21.15.73:0/3214227118 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc086ffd640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0880ffe60 0x7fc08819d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.924+0000 7fc0867fc640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc088104ce0 0x7fc08819ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 -- 172.21.15.73:0/3214227118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0880ffe60 msgr2=0x7fc08819d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0880ffe60 0x7fc08819d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 -- 172.21.15.73:0/3214227118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc088104ce0 msgr2=0x7fc08819ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc088104ce0 0x7fc08819ddb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 -- 172.21.15.73:0/3214227118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc088198540 con 0x7fc088197a20 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc0877fe640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d750 tx=0x7fc08000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc08c9b1640 1 -- 172.21.15.73:0/3214227118 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080017070 con 0x7fc088197a20 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc08c9b1640 1 -- 172.21.15.73:0/3214227118 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc080005500 con 0x7fc088197a20 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc088198830 con 0x7fc088197a20 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc08c9b1640 1 -- 172.21.15.73:0/3214227118 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc080004d90 con 0x7fc088197a20 2024-03-09T00:08:19.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.925+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0881a4a80 con 0x7fc088197a20 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.927+0000 7fc08c9b1640 1 -- 172.21.15.73:0/3214227118 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 25) v1 ==== 137523+0+0 (secure 0 0 0) 0x7fc080004f30 con 0x7fc088197a20 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.927+0000 7fc09058f740 1 Processor -- start 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.927+0000 7fc09058f740 1 -- start start 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.927+0000 7fc08c9b1640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 0x7fc0700a4f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d13d720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d13dc60 0x55558d140050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d097e00 con 0x55558d135080 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d040b10 con 0x55558d134760 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc09058f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d093ea0 con 0x55558d13dc60 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc0867fc640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 0x7fc0700a4f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc08c9b1640 1 -- 172.21.15.73:0/3214227118 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc0800ced30 con 0x7fc088197a20 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:43606/0 (socket says 172.21.15.73:43606) 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc0867fc640 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d13d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 -- 172.21.15.73:0/969484380 learned_addr learned my addr 172.21.15.73:0/969484380 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc0877fe640 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d13dc60 0x55558d140050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 -- 172.21.15.73:0/969484380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d13dc60 msgr2=0x55558d140050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d13dc60 0x55558d140050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 -- 172.21.15.73:0/969484380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 msgr2=0x55558d13d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d13d720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 -- 172.21.15.73:0/969484380 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558ce767d0 con 0x55558d134760 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc0867fc640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 0x7fc0700a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc07c036040 tx=0x7fc07c02c5d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc086ffd640 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc0780103a0 tx=0x7fc078010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55759d24e36ab9f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc06dffb640 1 -- 172.21.15.73:0/969484380 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078016070 con 0x55558d134760 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc06dffb640 1 -- 172.21.15.73:0/969484380 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc078004d20 con 0x55558d134760 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.928+0000 7fc06dffb640 1 -- 172.21.15.73:0/969484380 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc078005440 con 0x55558d134760 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- 172.21.15.73:0/969484380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 msgr2=0x55558d134b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc0780103a0 tx=0x7fc078010860 comp rx=0 tx=0).stop 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- 172.21.15.73:0/969484380 shutdown_connections 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d13dc60 0x55558d140050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d13d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 --2- 172.21.15.73:0/969484380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d134b40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- 172.21.15.73:0/969484380 >> 172.21.15.73:0/969484380 conn(0x55558d12ee90 msgr2=0x55558d12f270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- 172.21.15.73:0/969484380 shutdown_connections 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- 172.21.15.73:0/969484380 wait complete. 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.929+0000 7fc09058f740 1 -- start start 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 Processor -- start 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d13d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 0x55558d13a790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d140a50 con 0x55558d13dc60 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d0540a0 con 0x55558d135080 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55558d140bd0 con 0x55558d134760 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc09058f740 10 client.0 _ll_register_callbacks cb 0x55558d053a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43088/0 (socket says 172.21.15.73:43088) 2024-03-09T00:08:19.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 -- 172.21.15.73:0/1577278463 learned_addr learned my addr 172.21.15.73:0/1577278463 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc086ffd640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0867fc640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 0x55558d13a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 -- 172.21.15.73:0/1577278463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 msgr2=0x55558d13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 -- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 msgr2=0x55558d13a790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 0x55558d13a790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0877fe640 1 -- 172.21.15.73:0/1577278463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55558cdac3f0 con 0x55558d13dc60 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.930+0000 7fc0867fc640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 0x55558d13a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc0877fe640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc080009610 tx=0x7fc08000de60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ead012225970a878 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08009ef00 con 0x55558d13dc60 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc0800974d0 con 0x55558d13dc60 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc078004d20 con 0x55558d13dc60 2024-03-09T00:08:19.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc0800a3550 con 0x55558d13dc60 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 10 client.15981 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55558cd6c0f0 con 0x55558d13dc60 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc078004d20 con 0x55558d13dc60 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc080096bb0 con 0x55558d13dc60 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 10 client.15981 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 10 client.15981 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 20 client.15981 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d14d6a0 0x55558d14fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 4 client.15981 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55558d0a5340 con 0x55558d14d6a0 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.931+0000 7fc086ffd640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d14d6a0 0x55558d14fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.932+0000 7fc066ffd640 10 client.15981 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.932+0000 7fc086ffd640 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d14d6a0 0x55558d14fab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=edbc41730dec8a3c server_cookie=5ec6a6cc0cc8446e in_seq=0 out_seq=0 2024-03-09T00:08:19.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.932+0000 7fc066ffd640 1 -- 172.21.15.73:0/1577278463 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55558d0a5340 con 0x55558d14d6a0 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.932+0000 7fc066ffd640 10 client.15981 handle_command_reply: tid=0 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.932+0000 7fc09058f740 1 client.15981 shutdown 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 msgr2=0x55558d13acd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 crypto rx=0x7fc080009610 tx=0x7fc08000de60 comp rx=0 tx=0).stop 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 shutdown_connections 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d14d6a0 0x55558d14fab0 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d13acd0 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d135080 0x55558d13a790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc066ffd640 0 client.15981 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 --2- 172.21.15.73:0/1577278463 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 >> 172.21.15.73:0/1577278463 conn(0x55558d12ee90 msgr2=0x55558d130be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 shutdown_connections 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 1 -- 172.21.15.73:0/1577278463 wait complete. 2024-03-09T00:08:19.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.933+0000 7fc09058f740 20 client.15981 trim_cache size 0 max 16384 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 Processor -- start 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 -- start start 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d13cec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 0x55558d13d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881ac650 con 0x55558d13dc60 2024-03-09T00:08:19.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881aec10 con 0x55558d134760 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc09058f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881acc50 con 0x55558d135080 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43096/0 (socket says 172.21.15.73:43096) 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 -- 172.21.15.73:0/2012545533 learned_addr learned my addr 172.21.15.73:0/2012545533 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc086ffd640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d13cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 -- 172.21.15.73:0/2012545533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 msgr2=0x55558d13d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0867fc640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 0x55558d13d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 0x55558d13d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 -- 172.21.15.73:0/2012545533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 msgr2=0x55558d13cec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d13cec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 -- 172.21.15.73:0/2012545533 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55558d138ce0 con 0x55558d13dc60 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0867fc640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 0x55558d13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.948+0000 7fc0877fe640 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d720 tx=0x7fc08009afe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9845627485727ae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc06d7fa640 1 -- 172.21.15.73:0/2012545533 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc0800cbeb0 con 0x55558d13dc60 2024-03-09T00:08:19.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc06d7fa640 1 -- 172.21.15.73:0/2012545533 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc08009e020 con 0x55558d13dc60 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc06d7fa640 1 -- 172.21.15.73:0/2012545533 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc08009a2b0 con 0x55558d13dc60 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 -- 172.21.15.73:0/2012545533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 msgr2=0x55558d144e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d720 tx=0x7fc08009afe0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 -- 172.21.15.73:0/2012545533 shutdown_connections 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d13dc60 0x55558d144e10 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d135080 0x55558d13d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 --2- 172.21.15.73:0/2012545533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d134760 0x55558d13cec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 -- 172.21.15.73:0/2012545533 >> 172.21.15.73:0/2012545533 conn(0x55558d12ee90 msgr2=0x55558d1498c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 -- 172.21.15.73:0/2012545533 shutdown_connections 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.949+0000 7fc09058f740 1 -- 172.21.15.73:0/2012545533 wait complete. 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 -- start start 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 Processor -- start 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d145310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d13dc60 0x55558d143e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881ad290 con 0x55558d135080 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881ae5b0 con 0x55558d13dc60 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0881ad8f0 con 0x55558d134760 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc09058f740 10 client.0 _ll_register_callbacks cb 0x7fc080097140 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:19.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43100/0 (socket says 172.21.15.73:43100) 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 -- 172.21.15.73:0/2874592956 learned_addr learned my addr 172.21.15.73:0/2874592956 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc086ffd640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d145310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0877fe640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d13dc60 0x55558d143e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 -- 172.21.15.73:0/2874592956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 msgr2=0x55558d145310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d145310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 -- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d13dc60 msgr2=0x55558d143e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d13dc60 0x55558d143e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.950+0000 7fc0867fc640 1 -- 172.21.15.73:0/2874592956 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55558d1a3bc0 con 0x55558d135080 2024-03-09T00:08:19.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc0867fc640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009900 tx=0x7fc07c005fd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf944c77c4025464 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc07c0050e0 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc07c005280 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc08009e020 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc07c062050 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 10 client.15987 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55558d1a4cf0 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc08009e020 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc07c003c60 con 0x55558d135080 2024-03-09T00:08:19.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 10 client.15987 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 10 client.15987 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 20 client.15987 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d1a5450 0x55558d1a7870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 4 client.15987 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55558d09f690 con 0x55558d1a5450 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.951+0000 7fc086ffd640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d1a5450 0x55558d1a7870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc066ffd640 10 client.15987 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc086ffd640 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d1a5450 0x55558d1a7870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca29e92ce272cdde server_cookie=83b0a01f7300ba3b in_seq=0 out_seq=0 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc066ffd640 1 -- 172.21.15.73:0/2874592956 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55558d09f690 con 0x55558d1a5450 2024-03-09T00:08:19.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc066ffd640 10 client.15987 handle_command_reply: tid=0 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 client.15987 shutdown 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 msgr2=0x55558d143700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 crypto rx=0x7fc07c009900 tx=0x7fc07c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 shutdown_connections 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55558d1a5450 0x55558d1a7870 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55558d13dc60 0x55558d143e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc066ffd640 0 client.15987 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55558d135080 0x55558d143700 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 --2- 172.21.15.73:0/2874592956 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55558d134760 0x55558d145310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 >> 172.21.15.73:0/2874592956 conn(0x55558d12ee90 msgr2=0x55558d13a750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 shutdown_connections 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 1 -- 172.21.15.73:0/2874592956 wait complete. 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.952+0000 7fc09058f740 20 client.15987 trim_cache size 0 max 16384 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-09T00:08:19.958 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: "2e51ff4d-21ea-4e42-bd20-1bf8f00dcc8c": { 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "2e51ff4d-21ea-4e42-bd20-1bf8f00dcc8c", 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:08:19.959 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 msgr2=0x7fc0700a4f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 0x7fc0700a4f50 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fc07c036040 tx=0x7fc07c02c5d0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 msgr2=0x7fc088197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 crypto rx=0x7fc08000d750 tx=0x7fc08000dc10 comp rx=0 tx=0).stop 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 shutdown_connections 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0700a2a90 0x7fc0700a4f50 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc088197a20 0x7fc088197ed0 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc088104ce0 0x7fc08819ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 --2- 172.21.15.73:0/3214227118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0880ffe60 0x7fc08819d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:19.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 >> 172.21.15.73:0/3214227118 conn(0x7fc088075450 msgr2=0x7fc088105e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:19.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 shutdown_connections 2024-03-09T00:08:19.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:19.953+0000 7fc08d9b3640 1 -- 172.21.15.73:0/3214227118 wait complete. 2024-03-09T00:08:19.962 INFO:tasks.cephfs.filesystem:scrub status for tag:2e51ff4d-21ea-4e42-bd20-1bf8f00dcc8c - {'path': '/', 'tag': '2e51ff4d-21ea-4e42-bd20-1bf8f00dcc8c', 'options': 'recursive,force'} 2024-03-09T00:08:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:20 smithi179 ceph-mon[36637]: pgmap v47: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:20 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:08:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:20 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:20 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:08:20.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:20 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:20 smithi073 ceph-mon[32309]: pgmap v47: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:20 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:08:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:20 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:20 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:08:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:20 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:20 smithi102 ceph-mon[36106]: pgmap v47: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:20 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:08:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:20 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:20 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:08:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:20 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:22.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:22 smithi179 ceph-mon[36637]: pgmap v48: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:22.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:22 smithi073 ceph-mon[32309]: pgmap v48: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:22.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:22 smithi102 ceph-mon[36106]: pgmap v48: 129 pgs: 129 active+clean; 15 MiB data, 422 MiB used, 1.0 TiB / 1.0 TiB avail; 195 KiB/s rd, 1.2 MiB/s wr, 22 op/s 2024-03-09T00:08:23.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:23 smithi179 ceph-mon[36637]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-09T00:08:23.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:23 smithi073 ceph-mon[32309]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-09T00:08:23.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:23 smithi102 ceph-mon[36106]: mgrmap e26: x(active, since 91s), standbys: y, z 2024-03-09T00:08:24.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:24 smithi179 ceph-mon[36637]: pgmap v49: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-03-09T00:08:24.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:24 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:08:24.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:24 smithi073 ceph-mon[32309]: pgmap v49: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-03-09T00:08:24.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:24 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:08:24.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:24 smithi102 ceph-mon[36106]: pgmap v49: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 409 KiB/s rd, 1.5 MiB/s wr, 33 op/s 2024-03-09T00:08:24.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:24 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 -- 172.21.15.73:0/189496553 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc105190 msgr2=0x7fcafc105570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 --2- 172.21.15.73:0/189496553 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc105190 0x7fcafc105570 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8009900 tx=0x7fcaf802bc30 comp rx=0 tx=0).stop 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 -- 172.21.15.73:0/189496553 shutdown_connections 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 --2- 172.21.15.73:0/189496553 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100e00 0x7fcafc10d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 --2- 172.21.15.73:0/189496553 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc100440 0x7fcafc1008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 --2- 172.21.15.73:0/189496553 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc105190 0x7fcafc105570 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 -- 172.21.15.73:0/189496553 >> 172.21.15.73:0/189496553 conn(0x7fcafc077040 msgr2=0x7fcafc077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 -- 172.21.15.73:0/189496553 shutdown_connections 2024-03-09T00:08:25.235 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.240+0000 7fcb036c0640 1 -- 172.21.15.73:0/189496553 wait complete. 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 Processor -- start 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 -- start start 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 0x7fcafc19c6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 0x7fcafc196890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc110630 con 0x7fcafc100e00 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1104b0 con 0x7fcafc100440 2024-03-09T00:08:25.237 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb036c0640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1107b0 con 0x7fcafc105190 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43106/0 (socket says 172.21.15.73:43106) 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2590695462 learned_addr learned my addr 172.21.15.73:0/2590695462 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 0x7fcafc19c6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 0x7fcafc196890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2590695462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 msgr2=0x7fcafc196890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 0x7fcafc196890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2590695462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 msgr2=0x7fcafc19c6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 0x7fcafc19c6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2590695462 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcafc197010 con 0x7fcafc100e00 2024-03-09T00:08:25.238 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 0x7fcafc196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:25.247 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.241+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 0x7fcafc19c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00d950 tx=0x7fcaec00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcaf27fc640 1 -- 172.21.15.73:0/2590695462 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec017030 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcafc197300 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcaf27fc640 1 -- 172.21.15.73:0/2590695462 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaec004020 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcaf27fc640 1 -- 172.21.15.73:0/2590695462 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec0047a0 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.242+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcafc1a35a0 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.244+0000 7fcaf27fc640 1 -- 172.21.15.73:0/2590695462 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcaec0041d0 con 0x7fcafc100e00 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 Processor -- start 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcaf27fc640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 0x7fcae80a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.248 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 -- start start 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 0x7fcae80a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcaf27fc640 1 -- 172.21.15.73:0/2590695462 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcaec0ced20 con 0x7fcafc100e00 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e16413a450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 0x55e164147b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16409e5f0 con 0x55e164139fd0 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e164047930 con 0x55e16413ad40 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0637e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16409a690 con 0x55e164145760 2024-03-09T00:08:25.249 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0145c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e16413a450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0145c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:43668/0 (socket says 172.21.15.73:43668) 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0145c640 1 -- 172.21.15.73:0/2601013955 learned_addr learned my addr 172.21.15.73:0/2601013955 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 0x55e164147b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2601013955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 msgr2=0x55e164147b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 0x55e164147b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2601013955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 msgr2=0x55e164139a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.250 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 -- 172.21.15.73:0/2601013955 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e163db3210 con 0x55e164139fd0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 0x55e164147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 0x7fcae80a4ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005fd0 tx=0x7fcaf40117b0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.245+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e16413a450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0cc080 tx=0x7fcaec09abc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a974decf49cf0fd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcae77fe640 1 -- 172.21.15.73:0/2601013955 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec09f0e0 con 0x55e164139fd0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcae77fe640 1 -- 172.21.15.73:0/2601013955 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaec0a3730 con 0x55e164139fd0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcae77fe640 1 -- 172.21.15.73:0/2601013955 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaec09a700 con 0x55e164139fd0 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 -- 172.21.15.73:0/2601013955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 msgr2=0x55e16413a450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e16413a450 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 crypto rx=0x7fcaec0cc080 tx=0x7fcaec09abc0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 -- 172.21.15.73:0/2601013955 shutdown_connections 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164145760 0x55e164147b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e16413a450 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.251 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2601013955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164139a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.246+0000 7fcb0637e740 1 -- 172.21.15.73:0/2601013955 >> 172.21.15.73:0/2601013955 conn(0x55e1641015b0 msgr2=0x55e164134cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- 172.21.15.73:0/2601013955 shutdown_connections 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- 172.21.15.73:0/2601013955 wait complete. 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- start start 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 Processor -- start 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e16413ad40 0x55e1641417f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 0x55e16414c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16413b8f0 con 0x55e164139fd0 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16405ad90 con 0x55e164145760 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e16413ba70 con 0x55e16413ad40 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.247+0000 7fcb0637e740 10 client.0 _ll_register_callbacks cb 0x55e163db2a00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.252 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb01c5d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 0x55e16414c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43130/0 (socket says 172.21.15.73:43130) 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 -- 172.21.15.73:0/2448912705 learned_addr learned my addr 172.21.15.73:0/2448912705 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e16413ad40 0x55e1641417f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 -- 172.21.15.73:0/2448912705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e16413ad40 msgr2=0x55e1641417f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e16413ad40 0x55e1641417f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 -- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 msgr2=0x55e16414c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 0x55e16414c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 -- 172.21.15.73:0/2448912705 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e163e7f620 con 0x55e164139fd0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 0x55e16414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8004270 tx=0x7fcaf80042a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b343527cc4ae9434 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcae4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf802c8b0 con 0x55e164139fd0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcaf802ca50 con 0x55e164139fd0 2024-03-09T00:08:25.253 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcaec0a3730 con 0x55e164139fd0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcaf802cc50 con 0x55e164139fd0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0637e740 10 client.15999 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.248+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e1641569f0 con 0x55e164139fd0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaec0a3730 con 0x55e164139fd0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcaf8003b40 con 0x55e164139fd0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 10 client.15999 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 10 client.15999 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 20 client.15999 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e164157eb0 0x55e16415a2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 4 client.15999 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e1640abb30 con 0x55e164157eb0 2024-03-09T00:08:25.254 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e164157eb0 0x55e16415a2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e164157eb0 0x55e16415a2a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9b1a66afd8302984 server_cookie=88d5b21ae7eba4b5 in_seq=0 out_seq=0 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.249+0000 7fcae4ff9640 10 client.15999 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcae4ff9640 1 -- 172.21.15.73:0/2448912705 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e1640abb30 con 0x55e164157eb0 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcae4ff9640 10 client.15999 handle_command_reply: tid=0 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 client.15999 shutdown 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 msgr2=0x55e1641412b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 crypto rx=0x7fcaf8004270 tx=0x7fcaf80042a0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 shutdown_connections 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e164157eb0 0x55e16415a2a0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.255 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e164145760 0x55e16414c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e16413ad40 0x55e1641417f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcae4ff9640 0 client.15999 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2448912705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164139fd0 0x55e1641412b0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 >> 172.21.15.73:0/2448912705 conn(0x55e1641015b0 msgr2=0x55e164136a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 shutdown_connections 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 1 -- 172.21.15.73:0/2448912705 wait complete. 2024-03-09T00:08:25.256 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.252+0000 7fcb0637e740 20 client.15999 trim_cache size 0 max 16384 2024-03-09T00:08:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 Processor -- start 2024-03-09T00:08:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 -- start start 2024-03-09T00:08:25.264 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e164141410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164151120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1aa470 con 0x55e164145760 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1aca30 con 0x55e16413ad40 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb0637e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1aaa70 con 0x55e164139fd0 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb01c5d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.265 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb01c5d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43146/0 (socket says 172.21.15.73:43146) 2024-03-09T00:08:25.266 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb01c5d640 1 -- 172.21.15.73:0/2528397468 learned_addr learned my addr 172.21.15.73:0/2528397468 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:25.266 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.265+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164151120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.266 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb0145c640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e164141410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.266 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 -- 172.21.15.73:0/2528397468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 msgr2=0x55e164141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.266 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e164141410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 -- 172.21.15.73:0/2528397468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 msgr2=0x55e164151120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164151120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 -- 172.21.15.73:0/2528397468 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e164147ec0 con 0x55e164145760 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4012c20 tx=0x7fcaf4016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=693c4b70c60a1435 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcae6ffd640 1 -- 172.21.15.73:0/2528397468 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4012e50 con 0x55e164145760 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcae6ffd640 1 -- 172.21.15.73:0/2528397468 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaf4025070 con 0x55e164145760 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcae6ffd640 1 -- 172.21.15.73:0/2528397468 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf40217a0 con 0x55e164145760 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.266+0000 7fcb0637e740 1 -- 172.21.15.73:0/2528397468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 msgr2=0x55e164151660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4012c20 tx=0x7fcaf4016000 comp rx=0 tx=0).stop 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- 172.21.15.73:0/2528397468 shutdown_connections 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e164151660 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e164151120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.267 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- 172.21.15.73:0/2528397468 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e164141410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- 172.21.15.73:0/2528397468 >> 172.21.15.73:0/2528397468 conn(0x55e1641015b0 msgr2=0x55e1641347f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- 172.21.15.73:0/2528397468 shutdown_connections 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- 172.21.15.73:0/2528397468 wait complete. 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- start start 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 Processor -- start 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e16414d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e16414d830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1ab0b0 con 0x55e164145760 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1ac3d0 con 0x55e16413ad40 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcafc1ab710 con 0x55e164139fd0 2024-03-09T00:08:25.268 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb0637e740 10 client.0 _ll_register_callbacks cb 0x7fcaec0051b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb01c5d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb01c5d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43162/0 (socket says 172.21.15.73:43162) 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.267+0000 7fcb01c5d640 1 -- 172.21.15.73:0/4008861709 learned_addr learned my addr 172.21.15.73:0/4008861709 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb00c5b640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e16414d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb0145c640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e16414d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 -- 172.21.15.73:0/4008861709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 msgr2=0x55e16414d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e16414d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 -- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 msgr2=0x55e16414d830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e16414d830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 -- 172.21.15.73:0/4008861709 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e1641ad900 con 0x55e164145760 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb01c5d640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcaf402aec0 tx=0x7fcaf402aa40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efc9b7da8967bcc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:25.270 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcaf4038060 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcaf40142e0 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcaf4025070 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcaf403f050 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb0637e740 10 client.16005 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e1641aeb80 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcaf4025070 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.268+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcaf4021890 con 0x55e164145760 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 10 client.16005 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 10 client.16005 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 20 client.16005 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e1641af1a0 0x55e1641b15c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 4 client.16005 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:08:25.271 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e1640a5e80 con 0x55e1641af1a0 2024-03-09T00:08:25.272 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0145c640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e1641af1a0 0x55e1641b15c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:25.272 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcae4ff9640 10 client.16005 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:25.272 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:25.269+0000 7fcb0145c640 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e1641af1a0 0x55e1641b15c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5320a2603a87ad43 server_cookie=bc0299b82d6ca6e0 in_seq=0 out_seq=0 2024-03-09T00:08:25.438 INFO:tasks.workunit.client.0.smithi073.stderr:Updating files: 7% (321/4261) Updating files: 8% (341/4261) Updating files: 9% (384/4261) Updating files: 10% (427/4261) Updating files: 11% (469/4261) Updating files: 12% (512/4261) Updating files: 13% (554/4261) Updating files: 14% (597/4261) Updating files: 15% (640/4261) Updating files: 16% (682/4261) Updating files: 17% (725/4261) Updating files: 17% (742/4261) Updating files: 18% (767/4261) Updating files: 19% (810/4261) Updating files: 20% (853/4261) Updating files: 21% (895/4261) Updating files: 22% (938/4261) Updating files: 23% (981/4261) Updating files: 24% (1023/4261) Updating files: 25% (1066/4261) Updating files: 26% (1108/4261) Updating files: 27% (1151/4261) Updating files: 28% (1194/4261) Updating files: 29% (1236/4261) Updating files: 30% (1279/4261) Updating files: 31% (1321/4261) Updating files: 32% (1364/4261) Updating files: 33% (1407/4261) Updating files: 34% (1449/4261) Updating files: 35% (1492/4261) Updating files: 35% (1501/4261) Updating files: 35% (1502/4261) Updating files: 36% (1534/4261) Updating files: 37% (1577/4261) Updating files: 38% (1620/4261) Updating files: 39% (1662/4261) Updating files: 40% (1705/4261) Updating files: 41% (1748/4261) Updating files: 42% (1790/4261) Updating files: 43% (1833/4261) Updating files: 44% (1875/4261) Updating files: 45% (1918/4261) Updating files: 46% (1961/4261) Updating files: 47% (2003/4261) Updating files: 47% (2005/4261) Updating files: 48% (2046/4261) Updating files: 49% (2088/4261) Updating files: 50% (2131/4261) Updating files: 51% (2174/4261) Updating files: 52% (2216/4261) Updating files: 53% (2259/4261) Updating files: 54% (2301/4261) Updating files: 55% (2344/4261) Updating files: 56% (2387/4261) Updating files: 57% (2429/4261) Updating files: 58% (2472/4261) Updating files: 58% (2509/4261) Updating files: 59% (2514/4261) Updating files: 60% (2557/4261) Updating files: 61% (2600/4261) Updating files: 62% (2642/4261) Updating files: 63% (2685/4261) Updating files: 64% (2728/4261) Updating files: 65% (2770/4261) Updating files: 66% (2813/4261) Updating files: 67% (2855/4261) Updating files: 68% (2898/4261) Updating files: 69% (2941/4261) Updating files: 70% (2983/4261) Updating files: 71% (3026/4261) Updating files: 72% (3068/4261) Updating files: 73% (3111/4261) Updating files: 74% (3154/4261) Updating files: 75% (3196/4261) Updating files: 76% (3239/4261) Updating files: 76% (3277/4261) Updating files: 77% (3281/4261) Updating files: 78% (3324/4261) Updating files: 79% (3367/4261) Updating files: 80% (3409/4261) Updating files: 81% (3452/4261) Updating files: 82% (3495/4261) Updating files: 83% (3537/4261) Updating files: 84% (3580/4261) Updating files: 85% (3622/4261) Updating files: 86% (3665/4261) Updating files: 87% (3708/4261) Updating files: 88% (3750/4261) Updating files: 89% (3793/4261) Updating files: 90% (3835/4261) Updating files: 91% (3878/4261) Updating files: 92% (3921/4261) Updating files: 93% (3963/4261) Updating files: 94% (4006/4261) Updating files: 95% (4048/4261) Updating files: 96% (4091/4261) Updating files: 97% (4134/4261) Updating files: 98% (4176/4261) Updating files: 99% (4219/4261) Updating files: 100% (4261/4261) Updating files: 100% (4261/4261), done. 2024-03-09T00:08:25.449 INFO:tasks.workunit.client.0.smithi073.stderr:+ cd xfstests-dev 2024-03-09T00:08:25.450 INFO:tasks.workunit.client.0.smithi073.stderr:+ git checkout 12973fc04fd10d4af086901e10ffa8e48866b735 2024-03-09T00:08:26.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:26 smithi179 ceph-mon[36637]: pgmap v50: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-03-09T00:08:26.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:26 smithi073 ceph-mon[32309]: pgmap v50: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-03-09T00:08:26.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:26 smithi102 ceph-mon[36106]: pgmap v50: 129 pgs: 129 active+clean; 19 MiB data, 431 MiB used, 1.0 TiB / 1.0 TiB avail; 397 KiB/s rd, 1.5 MiB/s wr, 31 op/s 2024-03-09T00:08:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcae4ff9640 1 -- 172.21.15.73:0/4008861709 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e1640a5e80 con 0x55e1641af1a0 2024-03-09T00:08:26.966 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcae4ff9640 10 client.16005 handle_command_reply: tid=0 2024-03-09T00:08:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 client.16005 shutdown 2024-03-09T00:08:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 msgr2=0x55e1641ad3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 crypto rx=0x7fcaf402aec0 tx=0x7fcaf402aa40 comp rx=0 tx=0).stop 2024-03-09T00:08:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 shutdown_connections 2024-03-09T00:08:26.967 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e1641af1a0 0x55e1641b15c0 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e164145760 0x55e1641ad3c0 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e16413ad40 0x55e16414d830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcae4ff9640 0 client.16005 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 --2- 172.21.15.73:0/4008861709 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e164139fd0 0x55e16414d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 >> 172.21.15.73:0/4008861709 conn(0x55e1641015b0 msgr2=0x55e16413f3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:26.968 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 shutdown_connections 2024-03-09T00:08:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 1 -- 172.21.15.73:0/4008861709 wait complete. 2024-03-09T00:08:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.970+0000 7fcb0637e740 20 client.16005 trim_cache size 0 max 16384 2024-03-09T00:08:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:08:26.969 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:08:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:08:26.970 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:08:26.971 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 msgr2=0x7fcae80a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 0x7fcae80a4ed0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fcaf4005fd0 tx=0x7fcaf40117b0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 msgr2=0x7fcafc19cc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 crypto rx=0x7fcaec00d950 tx=0x7fcaec00de10 comp rx=0 tx=0).stop 2024-03-09T00:08:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 shutdown_connections 2024-03-09T00:08:26.972 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcafc105190 0x7fcafc196890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcae80a2a10 0x7fcae80a4ed0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcafc100e00 0x7fcafc19cc20 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 --2- 172.21.15.73:0/2590695462 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcafc100440 0x7fcafc19c6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 >> 172.21.15.73:0/2590695462 conn(0x7fcafc077040 msgr2=0x7fcafc0fe3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 shutdown_connections 2024-03-09T00:08:26.973 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:26.971+0000 7fcb036c0640 1 -- 172.21.15.73:0/2590695462 wait complete. 2024-03-09T00:08:28.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:28 smithi073 ceph-mon[32309]: pgmap v51: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-03-09T00:08:28.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:28 smithi102 ceph-mon[36106]: pgmap v51: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-03-09T00:08:28.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:28 smithi179 ceph-mon[36637]: pgmap v51: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 884 KiB/s rd, 2.1 MiB/s wr, 1.12k op/s 2024-03-09T00:08:30.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:30 smithi073 ceph-mon[32309]: pgmap v52: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:30.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:30 smithi102 ceph-mon[36106]: pgmap v52: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:30 smithi179 ceph-mon[36637]: pgmap v52: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:32.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:32 smithi073 ceph-mon[32309]: pgmap v53: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:32.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:32 smithi102 ceph-mon[36106]: pgmap v53: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:32.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:32 smithi179 ceph-mon[36637]: pgmap v53: 129 pgs: 129 active+clean; 16 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 702 KiB/s rd, 1.0 MiB/s wr, 1.10k op/s 2024-03-09T00:08:34.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:34 smithi073 ceph-mon[32309]: pgmap v54: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-09T00:08:34.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:34 smithi102 ceph-mon[36106]: pgmap v54: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-09T00:08:34.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:34 smithi179 ceph-mon[36637]: pgmap v54: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.6 MiB/s rd, 2.8 MiB/s wr, 1.33k op/s 2024-03-09T00:08:36.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:36 smithi073 ceph-mon[32309]: pgmap v55: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2024-03-09T00:08:36.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:36 smithi102 ceph-mon[36106]: pgmap v55: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2024-03-09T00:08:36.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:36 smithi179 ceph-mon[36637]: pgmap v55: 129 pgs: 129 active+clean; 35 MiB data, 523 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 MiB/s rd, 2.4 MiB/s wr, 1.32k op/s 2024-03-09T00:08:38.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:38 smithi073 ceph-mon[32309]: pgmap v56: 129 pgs: 129 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 1.55k op/s 2024-03-09T00:08:38.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:38 smithi102 ceph-mon[36106]: pgmap v56: 129 pgs: 129 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 1.55k op/s 2024-03-09T00:08:38.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:38 smithi179 ceph-mon[36637]: pgmap v56: 129 pgs: 129 active+clean; 44 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 MiB/s rd, 3.4 MiB/s wr, 1.55k op/s 2024-03-09T00:08:39.116 INFO:tasks.workunit.client.0.smithi073.stderr:Note: switching to '12973fc04fd10d4af086901e10ffa8e48866b735'. 2024-03-09T00:08:39.116 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.116 INFO:tasks.workunit.client.0.smithi073.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr:state without impacting any branches by switching back to a branch. 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: git switch -c 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr:Or undo this operation with: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.117 INFO:tasks.workunit.client.0.smithi073.stderr: git switch - 2024-03-09T00:08:39.118 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.118 INFO:tasks.workunit.client.0.smithi073.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-09T00:08:39.118 INFO:tasks.workunit.client.0.smithi073.stderr: 2024-03-09T00:08:39.118 INFO:tasks.workunit.client.0.smithi073.stderr:HEAD is now at 12973fc0 btrfs/022: Correct seq number in golden output 2024-03-09T00:08:39.118 INFO:tasks.workunit.client.0.smithi073.stderr:+ make -j4 2024-03-09T00:08:39.125 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize -cfi 2024-03-09T00:08:39.863 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: putting auxiliary files in '.'. 2024-03-09T00:08:39.863 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file './config.guess' 2024-03-09T00:08:39.866 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file './config.sub' 2024-03-09T00:08:39.870 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file './install-sh' 2024-03-09T00:08:39.873 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file './ltmain.sh' 2024-03-09T00:08:39.895 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'. 2024-03-09T00:08:39.895 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file 'm4/libtool.m4' 2024-03-09T00:08:39.899 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file 'm4/ltoptions.m4' 2024-03-09T00:08:39.904 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file 'm4/ltsugar.m4' 2024-03-09T00:08:39.908 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file 'm4/ltversion.m4' 2024-03-09T00:08:39.913 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: copying file 'm4/lt~obsolete.m4' 2024-03-09T00:08:39.919 INFO:tasks.workunit.client.0.smithi073.stdout:libtoolize: Consider adding '-I m4' to ACLOCAL_AMFLAGS in Makefile.am. 2024-03-09T00:08:39.920 INFO:tasks.workunit.client.0.smithi073.stdout:cp include/install-sh . 2024-03-09T00:08:39.956 INFO:tasks.workunit.client.0.smithi073.stdout:aclocal -I m4 2024-03-09T00:08:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 -- 172.21.15.73:0/1015898088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100e00 msgr2=0x7faa8c10d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 --2- 172.21.15.73:0/1015898088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100e00 0x7faa8c10d020 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 crypto rx=0x7faa84009ca0 tx=0x7faa8402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 -- 172.21.15.73:0/1015898088 shutdown_connections 2024-03-09T00:08:40.066 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 --2- 172.21.15.73:0/1015898088 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100e00 0x7faa8c10d020 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.069 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 --2- 172.21.15.73:0/1015898088 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c100440 0x7faa8c1008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 --2- 172.21.15.73:0/1015898088 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c105190 0x7faa8c105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 -- 172.21.15.73:0/1015898088 >> 172.21.15.73:0/1015898088 conn(0x7faa8c077040 msgr2=0x7faa8c077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 -- 172.21.15.73:0/1015898088 shutdown_connections 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 -- 172.21.15.73:0/1015898088 wait complete. 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.059+0000 7faa92122640 1 Processor -- start 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 -- start start 2024-03-09T00:08:40.070 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c100e00 0x7faa8c19cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c105190 0x7faa8c196870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c110630 con 0x7faa8c100440 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1104b0 con 0x7faa8c105190 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa92122640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1107b0 con 0x7faa8c100e00 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51306/0 (socket says 172.21.15.73:51306) 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 -- 172.21.15.73:0/1217464938 learned_addr learned my addr 172.21.15.73:0/1217464938 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:40.071 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8affd640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c100e00 0x7faa8c19cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8bfff640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c105190 0x7faa8c196870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 -- 172.21.15.73:0/1217464938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c100e00 msgr2=0x7faa8c19cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c100e00 0x7faa8c19cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 -- 172.21.15.73:0/1217464938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c105190 msgr2=0x7faa8c196870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c105190 0x7faa8c196870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.060+0000 7faa8b7fe640 1 -- 172.21.15.73:0/1217464938 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa8c196ff0 con 0x7faa8c100440 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7faa8000d950 tx=0x7faa8000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa88ff9640 1 -- 172.21.15.73:0/1217464938 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa80017030 con 0x7faa8c100440 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa8c1972e0 con 0x7faa8c100440 2024-03-09T00:08:40.072 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa88ff9640 1 -- 172.21.15.73:0/1217464938 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa80004020 con 0x7faa8c100440 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa88ff9640 1 -- 172.21.15.73:0/1217464938 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800047a0 con 0x7faa8c100440 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.061+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa8c1a3580 con 0x7faa8c100440 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.063+0000 7faa88ff9640 1 -- 172.21.15.73:0/1217464938 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7faa800041d0 con 0x7faa8c100440 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.063+0000 7faa94dde740 1 Processor -- start 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 -- start start 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ece6450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf3b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ec4a5f0 con 0x564a9ece6d40 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ebf3930 con 0x564a9ece5fd0 2024-03-09T00:08:40.073 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa94dde740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ec46690 con 0x564a9ecf1760 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa88ff9640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 0x7faa740a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51320/0 (socket says 172.21.15.73:51320) 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 -- 172.21.15.73:0/175920216 learned_addr learned my addr 172.21.15.73:0/175920216 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 0x7faa740a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ece6450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa88ff9640 1 -- 172.21.15.73:0/1217464938 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faa800ceb70 con 0x7faa8c100440 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8affd640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 -- 172.21.15.73:0/175920216 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 msgr2=0x564a9ecf3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 -- 172.21.15.73:0/175920216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 msgr2=0x564a9ece6450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.074 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ece6450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8bfff640 1 -- 172.21.15.73:0/175920216 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a9e95f210 con 0x564a9ece6d40 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.064+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ece6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa8bfff640 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faa84009ca0 tx=0x7faa84032f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8748af514cbd425 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa8bfff640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 0x7faa740a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa84036000 tx=0x7faa84042000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa71ffb640 1 -- 172.21.15.73:0/175920216 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84050070 con 0x564a9ece6d40 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa71ffb640 1 -- 172.21.15.73:0/175920216 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa84003e30 con 0x564a9ece6d40 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa71ffb640 1 -- 172.21.15.73:0/175920216 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa84039040 con 0x564a9ece6d40 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 -- 172.21.15.73:0/175920216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 msgr2=0x564a9ece5a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7faa84009ca0 tx=0x7faa84032f60 comp rx=0 tx=0).stop 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 -- 172.21.15.73:0/175920216 shutdown_connections 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.075 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ece6450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 --2- 172.21.15.73:0/175920216 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ece5a90 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.065+0000 7faa94dde740 1 -- 172.21.15.73:0/175920216 >> 172.21.15.73:0/175920216 conn(0x564a9ecad5b0 msgr2=0x564a9ece0cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- 172.21.15.73:0/175920216 shutdown_connections 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- 172.21.15.73:0/175920216 wait complete. 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- start start 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 Processor -- start 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ece5fd0 0x564a9ecee7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ecf1760 0x564a9ecef250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ecf4430 con 0x564a9ece6d40 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ec06d90 con 0x564a9ecf1760 2024-03-09T00:08:40.076 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564a9ecf45b0 con 0x564a9ece5fd0 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa94dde740 10 client.0 _ll_register_callbacks cb 0x564a9e95ea00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa8affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa8affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51328/0 (socket says 172.21.15.73:51328) 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa8affd640 1 -- 172.21.15.73:0/2763099922 learned_addr learned my addr 172.21.15.73:0/2763099922 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.066+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ece5fd0 0x564a9ecee7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8bfff640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ecf1760 0x564a9ecef250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 -- 172.21.15.73:0/2763099922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ece5fd0 msgr2=0x564a9ecee7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ece5fd0 0x564a9ecee7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 -- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ecf1760 msgr2=0x564a9ecef250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ecf1760 0x564a9ecef250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 -- 172.21.15.73:0/2763099922 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a9ea2b620 con 0x564a9ece6d40 2024-03-09T00:08:40.077 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa8affd640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7faa7c012700 tx=0x7faa7c012b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e686c9a69c2a3c3a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c004050 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faa7c0041f0 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa84003e30 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa7c025050 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa94dde740 10 client.16017 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564a9ed007c0 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa84003e30 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.067+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faa7c00f620 con 0x564a9ece6d40 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 10 client.16017 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:40.078 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 10 client.16017 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 20 client.16017 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed01c80 0x564a9ed04070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 4 client.16017 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564a9ec57b30 con 0x564a9ed01c80 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed01c80 0x564a9ed04070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa6affd640 10 client.16017 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.068+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed01c80 0x564a9ed04070 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25debd7b6d59a258 server_cookie=bfa06355f7348a9c in_seq=0 out_seq=0 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa6affd640 1 -- 172.21.15.73:0/2763099922 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564a9ec57b30 con 0x564a9ed01c80 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa6affd640 10 client.16017 handle_command_reply: tid=0 2024-03-09T00:08:40.079 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 client.16017 shutdown 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 msgr2=0x564a9eceed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 crypto rx=0x7faa7c012700 tx=0x7faa7c012b00 comp rx=0 tx=0).stop 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 shutdown_connections 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed01c80 0x564a9ed04070 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ecf1760 0x564a9ecef250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9eceed10 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 --2- 172.21.15.73:0/2763099922 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ece5fd0 0x564a9ecee7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa6affd640 0 client.16017 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 >> 172.21.15.73:0/2763099922 conn(0x564a9ecad5b0 msgr2=0x564a9ece2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 shutdown_connections 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 1 -- 172.21.15.73:0/2763099922 wait complete. 2024-03-09T00:08:40.080 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.069+0000 7faa94dde740 20 client.16017 trim_cache size 0 max 16384 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.095+0000 7faa94dde740 1 Processor -- start 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.095+0000 7faa94dde740 1 -- start start 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 0x564a9ecf8820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf8d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1aa500 con 0x564a9ece5fd0 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1acac0 con 0x564a9ece6d40 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa94dde740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1aab00 con 0x564a9ecf1760 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.095 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51342/0 (socket says 172.21.15.73:51342) 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 -- 172.21.15.73:0/3335182153 learned_addr learned my addr 172.21.15.73:0/3335182153 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8affd640 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 0x564a9ecf8820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 -- 172.21.15.73:0/3335182153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 msgr2=0x564a9ecf8d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf8d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 -- 172.21.15.73:0/3335182153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 msgr2=0x564a9ecf8820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 0x564a9ecf8820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 -- 172.21.15.73:0/3335182153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564a9ece52a0 con 0x564a9ece5fd0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8affd640 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 0x564a9ecf8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7faa800cbf30 tx=0x7faa80016e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a95ceddd82bd048 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa717fa640 1 -- 172.21.15.73:0/3335182153 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa8009a740 con 0x564a9ece5fd0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa717fa640 1 -- 172.21.15.73:0/3335182153 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa8009b4e0 con 0x564a9ece5fd0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.096+0000 7faa717fa640 1 -- 172.21.15.73:0/3335182153 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa800171d0 con 0x564a9ece5fd0 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- 172.21.15.73:0/3335182153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 msgr2=0x564a9ecf0f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.096 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 crypto rx=0x7faa800cbf30 tx=0x7faa80016e50 comp rx=0 tx=0).stop 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- 172.21.15.73:0/3335182153 shutdown_connections 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf8d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece6d40 0x564a9ecf8820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 --2- 172.21.15.73:0/3335182153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece5fd0 0x564a9ecf0f00 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- 172.21.15.73:0/3335182153 >> 172.21.15.73:0/3335182153 conn(0x564a9ecad5b0 msgr2=0x564a9ece07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- 172.21.15.73:0/3335182153 shutdown_connections 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- 172.21.15.73:0/3335182153 wait complete. 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.097+0000 7faa94dde740 1 -- start start 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 Processor -- start 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ecf5490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf5f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ab140 con 0x564a9ece6d40 2024-03-09T00:08:40.097 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ac460 con 0x564a9ece5fd0 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa8c1ab7a0 con 0x564a9ecf1760 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa94dde740 10 client.0 _ll_register_callbacks cb 0x7faa84037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:51344/0 (socket says 172.21.15.73:51344) 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 -- 172.21.15.73:0/409761415 learned_addr learned my addr 172.21.15.73:0/409761415 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8bfff640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf5f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ecf5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 -- 172.21.15.73:0/409761415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 msgr2=0x564a9ecf5f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf5f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 -- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 msgr2=0x564a9ecf5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ecf5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.098 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 -- 172.21.15.73:0/409761415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564a9ece2fc0 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa8affd640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x564a9ecefda0 tx=0x7faa7c00f620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd8bdace1b685eef server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.098+0000 7faa6affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faa7c02d020 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7faa7c022070 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7faa8009b4e0 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7faa7c038050 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 10 client.16023 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564a9ed594c0 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faa8009b4e0 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7faa7c003680 con 0x564a9ece6d40 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 10 client.16023 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 10 client.16023 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 20 client.16023 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:40.099 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed59b80 0x564a9ed5bfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 4 client.16023 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-09T00:08:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564a9ec51e80 con 0x564a9ed59b80 2024-03-09T00:08:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.099+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed59b80 0x564a9ed5bfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.100+0000 7faa6affd640 10 client.16023 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:40.100 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.100+0000 7faa8b7fe640 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed59b80 0x564a9ed5bfa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fc57b3a4524af007 server_cookie=7d5f3e0bfa4aa15 in_seq=0 out_seq=0 2024-03-09T00:08:40.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:40 smithi073 ceph-mon[32309]: pgmap v57: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:40.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:40 smithi102 ceph-mon[36106]: pgmap v57: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:40.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:40 smithi179 ceph-mon[36637]: pgmap v57: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.937+0000 7faa6affd640 1 -- 172.21.15.73:0/409761415 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564a9ec51e80 con 0x564a9ed59b80 2024-03-09T00:08:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.937+0000 7faa6affd640 10 client.16023 handle_command_reply: tid=0 2024-03-09T00:08:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.937+0000 7faa94dde740 1 client.16023 shutdown 2024-03-09T00:08:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 msgr2=0x564a9ecf59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.933 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 crypto rx=0x564a9ecefda0 tx=0x7faa7c00f620 comp rx=0 tx=0).stop 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 shutdown_connections 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564a9ed59b80 0x564a9ed5bfa0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564a9ecf1760 0x564a9ecf5f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564a9ece6d40 0x564a9ecf59d0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 --2- 172.21.15.73:0/409761415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564a9ece5fd0 0x564a9ecf5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 >> 172.21.15.73:0/409761415 conn(0x564a9ecad5b0 msgr2=0x564a9ece1fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa6affd640 0 client.16023 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:40.934 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 shutdown_connections 2024-03-09T00:08:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 1 -- 172.21.15.73:0/409761415 wait complete. 2024-03-09T00:08:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.938+0000 7faa94dde740 20 client.16023 trim_cache size 0 max 16384 2024-03-09T00:08:40.935 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:{ 2024-03-09T00:08:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "message": "", 2024-03-09T00:08:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout: "return_code": 0 2024-03-09T00:08:40.936 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stdout:} 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 msgr2=0x7faa740a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 0x7faa740a4ed0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7faa84036000 tx=0x7faa84042000 comp rx=0 tx=0).stop 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 msgr2=0x7faa8c19c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 crypto rx=0x7faa8000d950 tx=0x7faa8000de10 comp rx=0 tx=0).stop 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 shutdown_connections 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7faa8c105190 0x7faa8c196870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7faa8c100e00 0x7faa8c19cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.937 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7faa740a2a10 0x7faa740a4ed0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 --2- 172.21.15.73:0/1217464938 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7faa8c100440 0x7faa8c19c6c0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 >> 172.21.15.73:0/1217464938 conn(0x7faa8c077040 msgr2=0x7faa8c0fe3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 shutdown_connections 2024-03-09T00:08:40.938 INFO:teuthology.task.background_exec.ubuntu@smithi073.front.sepia.ceph.com.smithi073.stderr:2024-03-09T00:08:40.939+0000 7faa92122640 1 -- 172.21.15.73:0/1217464938 wait complete. 2024-03-09T00:08:41.321 INFO:tasks.workunit.client.0.smithi073.stdout:autoheader 2024-03-09T00:08:41.617 INFO:tasks.workunit.client.0.smithi073.stdout:autoconf 2024-03-09T00:08:41.789 INFO:tasks.workunit.client.0.smithi073.stdout:./configure \ 2024-03-09T00:08:41.789 INFO:tasks.workunit.client.0.smithi073.stdout: --libexecdir=/usr/lib \ 2024-03-09T00:08:41.789 INFO:tasks.workunit.client.0.smithi073.stdout: --exec_prefix=/var/lib 2024-03-09T00:08:41.907 INFO:tasks.workunit.client.0.smithi073.stdout:checking build system type... x86_64-pc-linux-gnu 2024-03-09T00:08:41.907 INFO:tasks.workunit.client.0.smithi073.stdout:checking host system type... x86_64-pc-linux-gnu 2024-03-09T00:08:41.908 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to print strings... printf 2024-03-09T00:08:41.908 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gcc... gcc 2024-03-09T00:08:41.984 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether the C compiler works... yes 2024-03-09T00:08:41.984 INFO:tasks.workunit.client.0.smithi073.stdout:checking for C compiler default output file name... a.out 2024-03-09T00:08:42.022 INFO:tasks.workunit.client.0.smithi073.stdout:checking for suffix of executables... 2024-03-09T00:08:42.057 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether we are cross compiling... no 2024-03-09T00:08:42.091 INFO:tasks.workunit.client.0.smithi073.stdout:checking for suffix of object files... o 2024-03-09T00:08:42.118 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether we are using the GNU C compiler... yes 2024-03-09T00:08:42.135 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether gcc accepts -g... yes 2024-03-09T00:08:42.198 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gcc option to accept ISO C89... none needed 2024-03-09T00:08:42.200 INFO:tasks.workunit.client.0.smithi073.stdout:checking for a sed that does not truncate output... /usr/bin/sed 2024-03-09T00:08:42.205 INFO:tasks.workunit.client.0.smithi073.stdout:checking for grep that handles long lines and -e... /usr/bin/grep 2024-03-09T00:08:42.205 INFO:tasks.workunit.client.0.smithi073.stdout:checking for egrep... /usr/bin/grep -E 2024-03-09T00:08:42.206 INFO:tasks.workunit.client.0.smithi073.stdout:checking for fgrep... /usr/bin/grep -F 2024-03-09T00:08:42.209 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ld used by gcc... /usr/bin/ld 2024-03-09T00:08:42.211 INFO:tasks.workunit.client.0.smithi073.stdout:checking if the linker (/usr/bin/ld) is GNU ld... yes 2024-03-09T00:08:42.222 INFO:tasks.workunit.client.0.smithi073.stdout:checking for BSD- or MS-compatible name lister (nm)... /usr/bin/nm -B 2024-03-09T00:08:42.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:42 smithi073 ceph-mon[32309]: pgmap v58: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:42.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:42 smithi102 ceph-mon[36106]: pgmap v58: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:42.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:42 smithi179 ceph-mon[36637]: pgmap v58: 129 pgs: 129 active+clean; 50 MiB data, 595 MiB used, 1.0 TiB / 1.0 TiB avail; 1.9 MiB/s rd, 3.5 MiB/s wr, 735 op/s 2024-03-09T00:08:43.191 INFO:tasks.workunit.client.0.smithi073.stdout:checking the name lister (/usr/bin/nm -B) interface... BSD nm 2024-03-09T00:08:43.191 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether ln -s works... yes 2024-03-09T00:08:43.228 INFO:tasks.workunit.client.0.smithi073.stdout:checking the maximum length of command line arguments... 1572864 2024-03-09T00:08:43.231 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to convert x86_64-pc-linux-gnu file names to x86_64-pc-linux-gnu format... func_convert_file_noop 2024-03-09T00:08:43.232 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to convert x86_64-pc-linux-gnu file names to toolchain format... func_convert_file_noop 2024-03-09T00:08:43.232 INFO:tasks.workunit.client.0.smithi073.stdout:checking for /usr/bin/ld option to reload object files... -r 2024-03-09T00:08:43.232 INFO:tasks.workunit.client.0.smithi073.stdout:checking for objdump... objdump 2024-03-09T00:08:43.233 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to recognize dependent libraries... pass_all 2024-03-09T00:08:43.233 INFO:tasks.workunit.client.0.smithi073.stdout:checking for dlltool... no 2024-03-09T00:08:43.233 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to associate runtime and link libraries... printf %s\n 2024-03-09T00:08:43.233 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ar... ar 2024-03-09T00:08:43.307 INFO:tasks.workunit.client.0.smithi073.stdout:checking for archiver @FILE support... @ 2024-03-09T00:08:43.307 INFO:tasks.workunit.client.0.smithi073.stdout:checking for strip... strip 2024-03-09T00:08:43.307 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ranlib... ranlib 2024-03-09T00:08:43.307 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gawk... gawk 2024-03-09T00:08:43.405 INFO:tasks.workunit.client.0.smithi073.stdout:checking command to parse /usr/bin/nm -B output from gcc object... ok 2024-03-09T00:08:43.406 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sysroot... no 2024-03-09T00:08:43.415 INFO:tasks.workunit.client.0.smithi073.stdout:checking for a working dd... /usr/bin/dd 2024-03-09T00:08:43.422 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to truncate binary pipes... /usr/bin/dd bs=4096 count=1 2024-03-09T00:08:43.452 INFO:tasks.workunit.client.0.smithi073.stdout:checking for mt... no 2024-03-09T00:08:43.463 INFO:tasks.workunit.client.0.smithi073.stdout:checking if : is a manifest tool... no 2024-03-09T00:08:43.794 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to run the C preprocessor... gcc -E 2024-03-09T00:08:43.970 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ANSI C header files... yes 2024-03-09T00:08:44.001 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/types.h... yes 2024-03-09T00:08:44.036 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/stat.h... yes 2024-03-09T00:08:44.071 INFO:tasks.workunit.client.0.smithi073.stdout:checking for stdlib.h... yes 2024-03-09T00:08:44.107 INFO:tasks.workunit.client.0.smithi073.stdout:checking for string.h... yes 2024-03-09T00:08:44.146 INFO:tasks.workunit.client.0.smithi073.stdout:checking for memory.h... yes 2024-03-09T00:08:44.182 INFO:tasks.workunit.client.0.smithi073.stdout:checking for strings.h... yes 2024-03-09T00:08:44.219 INFO:tasks.workunit.client.0.smithi073.stdout:checking for inttypes.h... yes 2024-03-09T00:08:44.256 INFO:tasks.workunit.client.0.smithi073.stdout:checking for stdint.h... yes 2024-03-09T00:08:44.295 INFO:tasks.workunit.client.0.smithi073.stdout:checking for unistd.h... yes 2024-03-09T00:08:44.339 INFO:tasks.workunit.client.0.smithi073.stdout:checking for dlfcn.h... yes 2024-03-09T00:08:44.345 INFO:tasks.workunit.client.0.smithi073.stdout:checking for objdir... .libs 2024-03-09T00:08:44.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:44 smithi073 ceph-mon[32309]: pgmap v59: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 1.44k op/s 2024-03-09T00:08:44.446 INFO:tasks.workunit.client.0.smithi073.stdout:checking if gcc supports -fno-rtti -fno-exceptions... no 2024-03-09T00:08:44.447 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gcc option to produce PIC... -fPIC -DPIC 2024-03-09T00:08:44.478 INFO:tasks.workunit.client.0.smithi073.stdout:checking if gcc PIC flag -fPIC -DPIC works... yes 2024-03-09T00:08:44.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:44 smithi102 ceph-mon[36106]: pgmap v59: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 1.44k op/s 2024-03-09T00:08:44.514 INFO:tasks.workunit.client.0.smithi073.stdout:checking if gcc static flag -static works... no 2024-03-09T00:08:44.567 INFO:tasks.workunit.client.0.smithi073.stdout:checking if gcc supports -c -o file.o... yes 2024-03-09T00:08:44.567 INFO:tasks.workunit.client.0.smithi073.stdout:checking if gcc supports -c -o file.o... (cached) yes 2024-03-09T00:08:44.573 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether the gcc linker (/usr/bin/ld -m elf_x86_64) supports shared libraries... yes 2024-03-09T00:08:44.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:44 smithi179 ceph-mon[36637]: pgmap v59: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 MiB/s rd, 4.2 MiB/s wr, 1.44k op/s 2024-03-09T00:08:44.621 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether -lc should be explicitly linked in... no 2024-03-09T00:08:44.685 INFO:tasks.workunit.client.0.smithi073.stdout:checking dynamic linker characteristics... GNU/Linux ld.so 2024-03-09T00:08:44.686 INFO:tasks.workunit.client.0.smithi073.stdout:checking how to hardcode library paths into programs... immediate 2024-03-09T00:08:44.687 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether stripping libraries is possible... yes 2024-03-09T00:08:44.687 INFO:tasks.workunit.client.0.smithi073.stdout:checking if libtool supports shared libraries... yes 2024-03-09T00:08:44.687 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether to build shared libraries... yes 2024-03-09T00:08:44.687 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether to build static libraries... yes 2024-03-09T00:08:44.693 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gcc... (cached) gcc 2024-03-09T00:08:44.720 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether we are using the GNU C compiler... (cached) yes 2024-03-09T00:08:44.720 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether gcc accepts -g... (cached) yes 2024-03-09T00:08:44.720 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gcc option to accept ISO C89... (cached) none needed 2024-03-09T00:08:44.720 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gmake... /usr/bin/gmake 2024-03-09T00:08:44.720 INFO:tasks.workunit.client.0.smithi073.stderr:./configure: line 11968: test: $(SHELL): binary operator expected 2024-03-09T00:08:44.721 INFO:tasks.workunit.client.0.smithi073.stdout:checking for tar... /bin/tar 2024-03-09T00:08:44.721 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gzip... /bin/gzip 2024-03-09T00:08:44.750 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether gcc -MM is supported... yes 2024-03-09T00:08:44.750 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sort... /bin/sort 2024-03-09T00:08:44.750 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether ln -s works... yes 2024-03-09T00:08:44.750 INFO:tasks.workunit.client.0.smithi073.stdout:checking for rpm... /bin/rpm 2024-03-09T00:08:44.754 INFO:tasks.workunit.client.0.smithi073.stdout:checking for rpmbuild... no 2024-03-09T00:08:44.754 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ANSI C header files... (cached) yes 2024-03-09T00:08:44.805 INFO:tasks.workunit.client.0.smithi073.stdout:checking assert.h usability... yes 2024-03-09T00:08:44.818 INFO:tasks.workunit.client.0.smithi073.stdout:checking assert.h presence... yes 2024-03-09T00:08:44.818 INFO:tasks.workunit.client.0.smithi073.stdout:checking for assert.h... yes 2024-03-09T00:08:44.869 INFO:tasks.workunit.client.0.smithi073.stdout:checking bstring.h usability... no 2024-03-09T00:08:44.895 INFO:tasks.workunit.client.0.smithi073.stdout:checking bstring.h presence... no 2024-03-09T00:08:44.895 INFO:tasks.workunit.client.0.smithi073.stdout:checking for bstring.h... no 2024-03-09T00:08:44.931 INFO:tasks.workunit.client.0.smithi073.stdout:checking libgen.h usability... yes 2024-03-09T00:08:44.944 INFO:tasks.workunit.client.0.smithi073.stdout:checking libgen.h presence... yes 2024-03-09T00:08:44.944 INFO:tasks.workunit.client.0.smithi073.stdout:checking for libgen.h... yes 2024-03-09T00:08:44.995 INFO:tasks.workunit.client.0.smithi073.stdout:checking dirent.h usability... yes 2024-03-09T00:08:45.009 INFO:tasks.workunit.client.0.smithi073.stdout:checking dirent.h presence... yes 2024-03-09T00:08:45.009 INFO:tasks.workunit.client.0.smithi073.stdout:checking for dirent.h... yes 2024-03-09T00:08:45.051 INFO:tasks.workunit.client.0.smithi073.stdout:checking errno.h usability... yes 2024-03-09T00:08:45.064 INFO:tasks.workunit.client.0.smithi073.stdout:checking errno.h presence... yes 2024-03-09T00:08:45.064 INFO:tasks.workunit.client.0.smithi073.stdout:checking for errno.h... yes 2024-03-09T00:08:45.116 INFO:tasks.workunit.client.0.smithi073.stdout:checking malloc.h usability... yes 2024-03-09T00:08:45.130 INFO:tasks.workunit.client.0.smithi073.stdout:checking malloc.h presence... yes 2024-03-09T00:08:45.130 INFO:tasks.workunit.client.0.smithi073.stdout:checking for malloc.h... yes 2024-03-09T00:08:45.176 INFO:tasks.workunit.client.0.smithi073.stdout:checking uuid.h usability... no 2024-03-09T00:08:45.199 INFO:tasks.workunit.client.0.smithi073.stdout:checking uuid.h presence... no 2024-03-09T00:08:45.199 INFO:tasks.workunit.client.0.smithi073.stdout:checking for uuid.h... no 2024-03-09T00:08:45.239 INFO:tasks.workunit.client.0.smithi073.stdout:checking uuid/uuid.h usability... yes 2024-03-09T00:08:45.254 INFO:tasks.workunit.client.0.smithi073.stdout:checking uuid/uuid.h presence... yes 2024-03-09T00:08:45.254 INFO:tasks.workunit.client.0.smithi073.stdout:checking for uuid/uuid.h... yes 2024-03-09T00:08:45.300 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/uuid.h usability... no 2024-03-09T00:08:45.322 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/uuid.h presence... no 2024-03-09T00:08:45.322 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/uuid.h... no 2024-03-09T00:08:45.360 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/file.h usability... yes 2024-03-09T00:08:45.373 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/file.h presence... yes 2024-03-09T00:08:45.373 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/file.h... yes 2024-03-09T00:08:45.425 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fcntl.h usability... yes 2024-03-09T00:08:45.438 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fcntl.h presence... yes 2024-03-09T00:08:45.438 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/fcntl.h... yes 2024-03-09T00:08:45.487 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/syssgi.h usability... no 2024-03-09T00:08:45.513 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/syssgi.h presence... no 2024-03-09T00:08:45.513 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/syssgi.h... no 2024-03-09T00:08:45.610 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/param.h usability... yes 2024-03-09T00:08:45.628 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/param.h presence... yes 2024-03-09T00:08:45.628 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/param.h... yes 2024-03-09T00:08:45.643 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/stat.h... (cached) yes 2024-03-09T00:08:45.685 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/statvfs.h usability... yes 2024-03-09T00:08:45.701 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/statvfs.h presence... yes 2024-03-09T00:08:45.701 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/statvfs.h... yes 2024-03-09T00:08:45.747 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/time.h usability... yes 2024-03-09T00:08:45.762 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/time.h presence... yes 2024-03-09T00:08:45.763 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/time.h... yes 2024-03-09T00:08:45.807 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/ioctl.h usability... yes 2024-03-09T00:08:45.823 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/ioctl.h presence... yes 2024-03-09T00:08:45.824 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/ioctl.h... yes 2024-03-09T00:08:45.875 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/wait.h usability... yes 2024-03-09T00:08:45.893 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/wait.h presence... yes 2024-03-09T00:08:45.893 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/wait.h... yes 2024-03-09T00:08:45.907 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/types.h... (cached) yes 2024-03-09T00:08:45.911 INFO:tasks.workunit.client.0.smithi073.stdout:checking for strings.h... (cached) yes 2024-03-09T00:08:45.946 INFO:tasks.workunit.client.0.smithi073.stdout:checking err.h usability... yes 2024-03-09T00:08:45.959 INFO:tasks.workunit.client.0.smithi073.stdout:checking err.h presence... yes 2024-03-09T00:08:45.959 INFO:tasks.workunit.client.0.smithi073.stdout:checking for err.h... yes 2024-03-09T00:08:46.000 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/falloc.h usability... yes 2024-03-09T00:08:46.015 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/falloc.h presence... yes 2024-03-09T00:08:46.015 INFO:tasks.workunit.client.0.smithi073.stdout:checking for linux/falloc.h... yes 2024-03-09T00:08:46.058 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fs/xfs_fsops.h usability... no 2024-03-09T00:08:46.082 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fs/xfs_fsops.h presence... no 2024-03-09T00:08:46.082 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/fs/xfs_fsops.h... no 2024-03-09T00:08:46.121 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fs/xfs_itable.h usability... no 2024-03-09T00:08:46.145 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/fs/xfs_itable.h presence... no 2024-03-09T00:08:46.145 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/fs/xfs_itable.h... no 2024-03-09T00:08:46.184 INFO:tasks.workunit.client.0.smithi073.stdout:checking xfs/platform_defs.h usability... no 2024-03-09T00:08:46.209 INFO:tasks.workunit.client.0.smithi073.stdout:checking xfs/platform_defs.h presence... no 2024-03-09T00:08:46.209 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/platform_defs.h... no 2024-03-09T00:08:46.248 INFO:tasks.workunit.client.0.smithi073.stdout:checking btrfs/ioctl.h usability... no 2024-03-09T00:08:46.271 INFO:tasks.workunit.client.0.smithi073.stdout:checking btrfs/ioctl.h presence... no 2024-03-09T00:08:46.271 INFO:tasks.workunit.client.0.smithi073.stdout:checking for btrfs/ioctl.h... no 2024-03-09T00:08:46.311 INFO:tasks.workunit.client.0.smithi073.stdout:checking cifs/ioctl.h usability... no 2024-03-09T00:08:46.334 INFO:tasks.workunit.client.0.smithi073.stdout:checking cifs/ioctl.h presence... no 2024-03-09T00:08:46.334 INFO:tasks.workunit.client.0.smithi073.stdout:checking for cifs/ioctl.h... no 2024-03-09T00:08:46.371 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/mman.h usability... yes 2024-03-09T00:08:46.383 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:46 smithi179 ceph-mon[36637]: pgmap v60: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.21k op/s 2024-03-09T00:08:46.384 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/mman.h presence... yes 2024-03-09T00:08:46.384 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/mman.h... yes 2024-03-09T00:08:46.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:46 smithi073 ceph-mon[32309]: pgmap v60: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.21k op/s 2024-03-09T00:08:46.423 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/xfs_log_format.h... no 2024-03-09T00:08:46.467 INFO:tasks.workunit.client.0.smithi073.stdout:checking for uuid_compare... no 2024-03-09T00:08:46.502 INFO:tasks.workunit.client.0.smithi073.stdout:checking for uuid_compare in -luuid... yes 2024-03-09T00:08:46.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:46 smithi102 ceph-mon[36106]: pgmap v60: 129 pgs: 129 active+clean; 22 MiB data, 601 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.21k op/s 2024-03-09T00:08:46.557 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/xfs.h... yes 2024-03-09T00:08:46.585 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/libxfs.h... no 2024-03-09T00:08:46.611 INFO:tasks.workunit.client.0.smithi073.stdout:checking whether xlog_assign_lsn is declared... no 2024-03-09T00:08:46.645 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/xqm.h... yes 2024-03-09T00:08:46.693 INFO:tasks.workunit.client.0.smithi073.stdout:checking xfsctl from xfs/xfs.h... ok 2024-03-09T00:08:46.725 INFO:tasks.workunit.client.0.smithi073.stdout:checking xfs/handle.h usability... yes 2024-03-09T00:08:46.738 INFO:tasks.workunit.client.0.smithi073.stdout:checking xfs/handle.h presence... yes 2024-03-09T00:08:46.738 INFO:tasks.workunit.client.0.smithi073.stdout:checking for xfs/handle.h... yes 2024-03-09T00:08:46.775 INFO:tasks.workunit.client.0.smithi073.stdout:checking for attr_list_by_handle in -lhandle... yes 2024-03-09T00:08:46.833 INFO:tasks.workunit.client.0.smithi073.stdout:checking attr/attributes.h usability... yes 2024-03-09T00:08:46.845 INFO:tasks.workunit.client.0.smithi073.stdout:checking attr/attributes.h presence... yes 2024-03-09T00:08:46.846 INFO:tasks.workunit.client.0.smithi073.stdout:checking for attr/attributes.h... yes 2024-03-09T00:08:46.890 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/attributes.h usability... no 2024-03-09T00:08:46.913 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/attributes.h presence... no 2024-03-09T00:08:46.913 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/attributes.h... no 2024-03-09T00:08:46.949 INFO:tasks.workunit.client.0.smithi073.stdout:checking for attr_list in -lattr... yes 2024-03-09T00:08:46.985 INFO:tasks.workunit.client.0.smithi073.stdout:checking for attr_set in -lattr... yes 2024-03-09T00:08:47.014 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/acl.h usability... yes 2024-03-09T00:08:47.028 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/acl.h presence... yes 2024-03-09T00:08:47.028 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/acl.h... yes 2024-03-09T00:08:47.057 INFO:tasks.workunit.client.0.smithi073.stdout:checking acl/libacl.h usability... yes 2024-03-09T00:08:47.070 INFO:tasks.workunit.client.0.smithi073.stdout:checking acl/libacl.h presence... yes 2024-03-09T00:08:47.071 INFO:tasks.workunit.client.0.smithi073.stdout:checking for acl/libacl.h... yes 2024-03-09T00:08:47.107 INFO:tasks.workunit.client.0.smithi073.stdout:checking for acl_init in -lacl... yes 2024-03-09T00:08:47.160 INFO:tasks.workunit.client.0.smithi073.stdout:checking gdbm-ndbm.h usability... no 2024-03-09T00:08:47.183 INFO:tasks.workunit.client.0.smithi073.stdout:checking gdbm-ndbm.h presence... no 2024-03-09T00:08:47.184 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gdbm-ndbm.h... no 2024-03-09T00:08:47.217 INFO:tasks.workunit.client.0.smithi073.stdout:checking gdbm/ndbm.h usability... no 2024-03-09T00:08:47.239 INFO:tasks.workunit.client.0.smithi073.stdout:checking gdbm/ndbm.h presence... no 2024-03-09T00:08:47.240 INFO:tasks.workunit.client.0.smithi073.stdout:checking for gdbm/ndbm.h... no 2024-03-09T00:08:47.272 INFO:tasks.workunit.client.0.smithi073.stdout:checking ndbm.h usability... no 2024-03-09T00:08:47.294 INFO:tasks.workunit.client.0.smithi073.stdout:checking ndbm.h presence... no 2024-03-09T00:08:47.294 INFO:tasks.workunit.client.0.smithi073.stdout:checking for ndbm.h... no 2024-03-09T00:08:47.322 INFO:tasks.workunit.client.0.smithi073.stdout:checking libaio.h usability... yes 2024-03-09T00:08:47.337 INFO:tasks.workunit.client.0.smithi073.stdout:checking libaio.h presence... yes 2024-03-09T00:08:47.338 INFO:tasks.workunit.client.0.smithi073.stdout:checking for libaio.h... yes 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout:checking for dm_make_handle in -ldm... no 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout:WARNING: could not find a valid DMAPI base library. 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout:If you want DMAPI support please install the data migration 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout:API (dmapi) library package. Alternatively, run "make install" 2024-03-09T00:08:47.382 INFO:tasks.workunit.client.0.smithi073.stdout:from the dmapi source. 2024-03-09T00:08:47.383 INFO:tasks.workunit.client.0.smithi073.stdout: 2024-03-09T00:08:47.410 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/fiemap.h usability... yes 2024-03-09T00:08:47.422 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/fiemap.h presence... yes 2024-03-09T00:08:47.422 INFO:tasks.workunit.client.0.smithi073.stdout:checking for linux/fiemap.h... yes 2024-03-09T00:08:47.456 INFO:tasks.workunit.client.0.smithi073.stdout:checking for fallocate... yes 2024-03-09T00:08:47.498 INFO:tasks.workunit.client.0.smithi073.stdout:checking for open_by_handle_at... yes 2024-03-09T00:08:47.537 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/prctl.h usability... yes 2024-03-09T00:08:47.550 INFO:tasks.workunit.client.0.smithi073.stdout:checking sys/prctl.h presence... yes 2024-03-09T00:08:47.550 INFO:tasks.workunit.client.0.smithi073.stdout:checking for sys/prctl.h... yes 2024-03-09T00:08:47.579 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/fs.h usability... yes 2024-03-09T00:08:47.591 INFO:tasks.workunit.client.0.smithi073.stdout:checking linux/fs.h presence... yes 2024-03-09T00:08:47.591 INFO:tasks.workunit.client.0.smithi073.stdout:checking for linux/fs.h... yes 2024-03-09T00:08:47.645 INFO:tasks.workunit.client.0.smithi073.stdout:checking btrfsutil.h usability... no 2024-03-09T00:08:47.667 INFO:tasks.workunit.client.0.smithi073.stdout:checking btrfsutil.h presence... no 2024-03-09T00:08:47.667 INFO:tasks.workunit.client.0.smithi073.stdout:checking for btrfsutil.h... no 2024-03-09T00:08:47.704 INFO:tasks.workunit.client.0.smithi073.stdout:checking for copy_file_range... yes 2024-03-09T00:08:47.746 INFO:tasks.workunit.client.0.smithi073.stdout:checking for renameat2... yes 2024-03-09T00:08:47.763 INFO:tasks.workunit.client.0.smithi073.stdout:configure: creating ./config.status 2024-03-09T00:08:48.093 INFO:tasks.workunit.client.0.smithi073.stdout:config.status: creating include/builddefs 2024-03-09T00:08:48.105 INFO:tasks.workunit.client.0.smithi073.stdout:config.status: creating include/config.h 2024-03-09T00:08:48.114 INFO:tasks.workunit.client.0.smithi073.stdout:config.status: executing libtool commands 2024-03-09T00:08:48.183 INFO:tasks.workunit.client.0.smithi073.stdout:Building include 2024-03-09T00:08:48.183 INFO:tasks.workunit.client.0.smithi073.stdout:Building lib 2024-03-09T00:08:48.183 INFO:tasks.workunit.client.0.smithi073.stdout:Building tests 2024-03-09T00:08:48.188 INFO:tasks.workunit.client.0.smithi073.stdout: [LTDEP] 2024-03-09T00:08:48.345 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] dataascii.lo 2024-03-09T00:08:48.345 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] databin.lo 2024-03-09T00:08:48.346 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] datapid.lo 2024-03-09T00:08:48.346 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] file_lock.lo 2024-03-09T00:08:48.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:48 smithi073 ceph-mon[32309]: pgmap v61: 129 pgs: 129 active+clean; 22 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.22k op/s 2024-03-09T00:08:48.453 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] forker.lo 2024-03-09T00:08:48.466 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] pattern.lo 2024-03-09T00:08:48.500 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] open_flags.lo 2024-03-09T00:08:48.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:48 smithi102 ceph-mon[36106]: pgmap v61: 129 pgs: 129 active+clean; 22 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.22k op/s 2024-03-09T00:08:48.513 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] random_range.lo 2024-03-09T00:08:48.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:48 smithi179 ceph-mon[36637]: pgmap v61: 129 pgs: 129 active+clean; 22 MiB data, 620 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 2.4 MiB/s wr, 1.22k op/s 2024-03-09T00:08:48.678 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] string_to_tokens.lo 2024-03-09T00:08:48.743 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] str_to_bytes.lo 2024-03-09T00:08:48.743 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] tlibio.lo 2024-03-09T00:08:48.770 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] write_log.lo 2024-03-09T00:08:48.874 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] random.lo 2024-03-09T00:08:49.078 INFO:tasks.workunit.client.0.smithi073.stdout: [LD] 2024-03-09T00:08:49.223 INFO:tasks.workunit.client.0.smithi073.stdout:Building ltp 2024-03-09T00:08:49.223 INFO:tasks.workunit.client.0.smithi073.stdout:Building src 2024-03-09T00:08:49.224 INFO:tasks.workunit.client.0.smithi073.stdout:Building m4 2024-03-09T00:08:49.224 INFO:tasks.workunit.client.0.smithi073.stdout:Building common 2024-03-09T00:08:49.229 INFO:tasks.workunit.client.0.smithi073.stdout: [DEP] 2024-03-09T00:08:49.242 INFO:tasks.workunit.client.0.smithi073.stdout: [DEP] 2024-03-09T00:08:49.421 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] doio 2024-03-09T00:08:49.421 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fsstress 2024-03-09T00:08:49.422 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fsx 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c: In function 'attr_list_path': 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:859:9: warning: 'attr_list' is deprecated: Use listxattr or llistxattr instead [-Wdeprecated-declarations] 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr: 859 | rval = attr_list(name->path, buffer, buffersize, flags, cursor); 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~ 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr:In file included from ../src/global.h:25, 2024-03-09T00:08:49.558 INFO:tasks.workunit.client.0.smithi073.stderr: from fsstress.c:12: 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:/usr/include/attr/attributes.h:165:12: note: declared here 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: 165 | extern int attr_list(const char *__path, char *__buffer, const int __buffersize, 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~ 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c: In function 'attr_remove_path': 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:878:9: warning: 'attr_remove' is deprecated: Use removexattr or lremovexattr instead [-Wdeprecated-declarations] 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: 878 | rval = attr_remove(name->path, attrname, flags); 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~ 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:In file included from ../src/global.h:25, 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: from fsstress.c:12: 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:/usr/include/attr/attributes.h:152:12: note: declared here 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: 152 | extern int attr_remove (const char *__path, const char *__attrname, 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~ 2024-03-09T00:08:49.559 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c: In function 'attr_set_path': 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:898:9: warning: 'attr_set' is deprecated: Use setxattr or lsetxattr instead [-Wdeprecated-declarations] 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr: 898 | rval = attr_set(name->path, attrname, attrvalue, valuelength, flags); 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~ 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr:In file included from ../src/global.h:25, 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr: from fsstress.c:12: 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr:/usr/include/attr/attributes.h:139:12: note: declared here 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr: 139 | extern int attr_set (const char *__path, const char *__attrname, 2024-03-09T00:08:49.560 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~ 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c: In function 'allocsp_f': 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:2047:32: error: 'XFS_IOC_ALLOCSP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr: 2047 | e = xfsctl(f.path, fd, XFS_IOC_ALLOCSP64, &fl) < 0 ? errno : 0; 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~~ 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr: | XFS_IOC_RESVSP64 2024-03-09T00:08:49.569 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:2047:32: note: each undeclared identifier is reported only once for each function it appears in 2024-03-09T00:08:49.594 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c: In function 'freesp_f': 2024-03-09T00:08:49.595 INFO:tasks.workunit.client.0.smithi073.stderr:fsstress.c:3640:32: error: 'XFS_IOC_FREESP64' undeclared (first use in this function); did you mean 'XFS_IOC_RESVSP64'? 2024-03-09T00:08:49.595 INFO:tasks.workunit.client.0.smithi073.stderr: 3640 | e = xfsctl(f.path, fd, XFS_IOC_FREESP64, &fl) < 0 ? errno : 0; 2024-03-09T00:08:49.595 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~ 2024-03-09T00:08:49.595 INFO:tasks.workunit.client.0.smithi073.stderr: | XFS_IOC_RESVSP64 2024-03-09T00:08:49.676 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[3]: *** [Makefile:47: fsstress] Error 1 2024-03-09T00:08:49.676 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-09T00:08:49.963 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:08:49.963 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr:doio.c: In function 'parse_cmdline': 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr:doio.c:3113:33: warning: '%s' directive argument is null [-Wformat-overflow=] 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr: 3113 | fprintf(stderr, 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~ 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr: 3114 | "%s%s: Illegal -C arg (%s). Must be one of: ", 2024-03-09T00:08:50.016 INFO:tasks.workunit.client.0.smithi073.stderr: | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:50.017 INFO:tasks.workunit.client.0.smithi073.stderr: 3115 | Prog, TagName, tok); 2024-03-09T00:08:50.017 INFO:tasks.workunit.client.0.smithi073.stderr: | ~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr:fsx.c: In function 'main': 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr:fsx.c:2662:33: warning: 'strncpy' specified bound 4096 equals destination size [-Wstringop-truncation] 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr: 2662 | strncpy(opsfile, optarg, sizeof(opsfile)); 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr:fsx.c:2632:25: warning: 'strncpy' specified bound 1024 equals destination size [-Wstringop-truncation] 2024-03-09T00:08:50.034 INFO:tasks.workunit.client.0.smithi073.stderr: 2632 | strncpy(dname, optarg, sizeof(dname)); 2024-03-09T00:08:50.035 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.080+0000 7fd584285640 1 -- 172.21.15.73:0/1303256332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102560 msgr2=0x7fd57c102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.080+0000 7fd584285640 1 --2- 172.21.15.73:0/1303256332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102560 0x7fd57c102940 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 crypto rx=0x7fd578009900 tx=0x7fd57802bcd0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 -- 172.21.15.73:0/1303256332 shutdown_connections 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 --2- 172.21.15.73:0/1303256332 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c10b930 0x7fd57c10dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 --2- 172.21.15.73:0/1303256332 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c102e80 0x7fd57c10b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 --2- 172.21.15.73:0/1303256332 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102560 0x7fd57c102940 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 -- 172.21.15.73:0/1303256332 >> 172.21.15.73:0/1303256332 conn(0x7fd57c077380 msgr2=0x7fd57c077790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 -- 172.21.15.73:0/1303256332 shutdown_connections 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 -- 172.21.15.73:0/1303256332 wait complete. 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 Processor -- start 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.081+0000 7fd584285640 1 -- start start 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c102560 0x7fd57c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c10b930 0x7fd57c197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c1116c0 con 0x7fd57c102e80 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c111540 con 0x7fd57c10b930 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd584285640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c111840 con 0x7fd57c102560 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37884/0 (socket says 172.21.15.73:37884) 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 -- 172.21.15.73:0/2431738651 learned_addr learned my addr 172.21.15.73:0/2431738651 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd582822640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c10b930 0x7fd57c197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd582021640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c102560 0x7fd57c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 -- 172.21.15.73:0/2431738651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c102560 msgr2=0x7fd57c19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c102560 0x7fd57c19d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 -- 172.21.15.73:0/2431738651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c10b930 msgr2=0x7fd57c197a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c10b930 0x7fd57c197a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.082+0000 7fd581820640 1 -- 172.21.15.73:0/2431738651 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd57c1981a0 con 0x7fd57c102e80 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd581820640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00e9f0 tx=0x7fd56c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd5737fe640 1 -- 172.21.15.73:0/2431738651 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c00cd80 con 0x7fd57c102e80 2024-03-09T00:08:50.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd57c198490 con 0x7fd57c102e80 2024-03-09T00:08:50.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd5737fe640 1 -- 172.21.15.73:0/2431738651 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd56c010910 con 0x7fd57c102e80 2024-03-09T00:08:50.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd5737fe640 1 -- 172.21.15.73:0/2431738651 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c0188f0 con 0x7fd57c102e80 2024-03-09T00:08:50.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.083+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd57c1a46e0 con 0x7fd57c102e80 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 Processor -- start 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 -- start start 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 0x55fa74c00720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c03050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74b5ae00 con 0x55fa74bf7760 2024-03-09T00:08:50.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74b03b10 con 0x55fa74c00c60 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd586e6f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74b56ea0 con 0x55fa74bf8080 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd582822640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd582822640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37896/0 (socket says 172.21.15.73:37896) 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.086+0000 7fd582822640 1 -- 172.21.15.73:0/2229577511 learned_addr learned my addr 172.21.15.73:0/2229577511 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582021640 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 0x55fa74c00720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd5737fe640 1 -- 172.21.15.73:0/2431738651 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd56c010ac0 con 0x7fd57c102e80 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 -- 172.21.15.73:0/2229577511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 msgr2=0x55fa74c00720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 0x55fa74c00720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 -- 172.21.15.73:0/2229577511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 msgr2=0x55fa74c03050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c03050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 -- 172.21.15.73:0/2229577511 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa749397d0 con 0x55fa74bf7760 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582021640 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 0x55fa74c00720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd582822640 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd574009f70 tx=0x7fd574010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9abe52d41fcb93 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd570bfe640 1 -- 172.21.15.73:0/2229577511 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574016070 con 0x55fa74bf7760 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd570bfe640 1 -- 172.21.15.73:0/2229577511 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd574004d20 con 0x55fa74bf7760 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd570bfe640 1 -- 172.21.15.73:0/2229577511 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574012950 con 0x55fa74bf7760 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.087+0000 7fd5737fe640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 0x7fd5680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.088+0000 7fd5737fe640 1 -- 172.21.15.73:0/2431738651 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd56c0cf640 con 0x7fd57c102e80 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.088+0000 7fd582021640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 0x7fd5680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.088+0000 7fd586e6f740 1 -- 172.21.15.73:0/2229577511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 msgr2=0x55fa74bf7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.088+0000 7fd582021640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 0x7fd5680a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd57802efd0 tx=0x7fd57802cf90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.088+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 crypto rx=0x7fd574009f70 tx=0x7fd574010800 comp rx=0 tx=0).stop 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 -- 172.21.15.73:0/2229577511 shutdown_connections 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c03050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf8080 0x55fa74c00720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2229577511 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74bf7b40 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 -- 172.21.15.73:0/2229577511 >> 172.21.15.73:0/2229577511 conn(0x55fa74bf1e90 msgr2=0x55fa74bf2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 -- 172.21.15.73:0/2229577511 shutdown_connections 2024-03-09T00:08:50.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 -- 172.21.15.73:0/2229577511 wait complete. 2024-03-09T00:08:50.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 -- start start 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 Processor -- start 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bfd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.089+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74c00c60 0x55fa74bfdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd586e6f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74c14a60 con 0x55fa74bf7760 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37904/0 (socket says 172.21.15.73:37904) 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 -- 172.21.15.73:0/2103418850 learned_addr learned my addr 172.21.15.73:0/2103418850 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74b170a0 con 0x55fa74bf8080 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fa74c14be0 con 0x55fa74c00c60 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd581820640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bfd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd586e6f740 10 client.0 _ll_register_callbacks cb 0x55fa74b16a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582822640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74c00c60 0x55fa74bfdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 -- 172.21.15.73:0/2103418850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74c00c60 msgr2=0x55fa74bfdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74c00c60 0x55fa74bfdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 -- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 msgr2=0x55fa74bfd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bfd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa7486f3f0 con 0x55fa74bf7760 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd581820640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bfd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd582021640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd57802c8c0 tx=0x7fd578009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1093f7b482c0385 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.090+0000 7fd565ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:50.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd578005110 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd5780052b0 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd574004d20 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd578004d10 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 10 client.16035 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fa7482f0f0 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd57804e070 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd578046030 con 0x55fa74bf7760 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 10 client.16035 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 10 client.16035 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 20 client.16035 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c23b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 4 client.16035 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.091+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd574004d20 con 0x55fa74c21730 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.092+0000 7fd581820640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c23b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.092+0000 7fd565ffb640 10 client.16035 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.092+0000 7fd581820640 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c23b20 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9e07c55ea841181b server_cookie=1ff99b187884c108 in_seq=0 out_seq=0 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd565ffb640 1 -- 172.21.15.73:0/2103418850 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fd574004d20 con 0x55fa74c21730 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd565ffb640 10 client.16035 handle_command_reply: tid=0 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 client.16035 shutdown 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 msgr2=0x55fa74c00420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 crypto rx=0x7fd57802c8c0 tx=0x7fd578009260 comp rx=0 tx=0).stop 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 shutdown_connections 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c23b20 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd565ffb640 0 client.16035 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74c00c60 0x55fa74bfdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bfd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2103418850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf7760 0x55fa74c00420 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 >> 172.21.15.73:0/2103418850 conn(0x55fa74bf1e90 msgr2=0x55fa74bf3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 shutdown_connections 2024-03-09T00:08:50.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 1 -- 172.21.15.73:0/2103418850 wait complete. 2024-03-09T00:08:50.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.093+0000 7fd586e6f740 20 client.16035 trim_cache size 0 max 16384 2024-03-09T00:08:50.124 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 Processor -- start 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 -- start start 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74bf8dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bffee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c070900 con 0x55fa74c00c60 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c072ec0 con 0x55fa74bf8080 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.123+0000 7fd586e6f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c070f00 con 0x55fa74bf7760 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37910/0 (socket says 172.21.15.73:37910) 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 -- 172.21.15.73:0/1285355516 learned_addr learned my addr 172.21.15.73:0/1285355516 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582021640 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74bf8dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd581820640 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bffee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 -- 172.21.15.73:0/1285355516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 msgr2=0x55fa74bf8dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74bf8dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 -- 172.21.15.73:0/1285355516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 msgr2=0x55fa74bffee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bffee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 -- 172.21.15.73:0/1285355516 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fa74bf7eb0 con 0x55fa74c00c60 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd582822640 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd5740124c0 tx=0x7fd574003a20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34438ea62d86e6e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd567fff640 1 -- 172.21.15.73:0/1285355516 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574016070 con 0x55fa74c00c60 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd567fff640 1 -- 172.21.15.73:0/1285355516 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5740058b0 con 0x55fa74c00c60 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.124+0000 7fd567fff640 1 -- 172.21.15.73:0/1285355516 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd574024440 con 0x55fa74c00c60 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- 172.21.15.73:0/1285355516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 msgr2=0x55fa74c00420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7fd5740124c0 tx=0x7fd574003a20 comp rx=0 tx=0).stop 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- 172.21.15.73:0/1285355516 shutdown_connections 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74c00c60 0x55fa74c00420 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74bf8080 0x55fa74bffee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 --2- 172.21.15.73:0/1285355516 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74bf8dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- 172.21.15.73:0/1285355516 >> 172.21.15.73:0/1285355516 conn(0x55fa74bf1e90 msgr2=0x55fa74bf86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- 172.21.15.73:0/1285355516 shutdown_connections 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- 172.21.15.73:0/1285355516 wait complete. 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 -- start start 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.125+0000 7fd586e6f740 1 Processor -- start 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74c18e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c17960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c071540 con 0x55fa74bf8080 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c072860 con 0x55fa74c00c60 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd57c071ba0 con 0x55fa74bf7760 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd586e6f740 10 client.0 _ll_register_callbacks cb 0x7fd578005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37922/0 (socket says 172.21.15.73:37922) 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 -- 172.21.15.73:0/2676387320 learned_addr learned my addr 172.21.15.73:0/2676387320 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd582822640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c17960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd582021640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74c18e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 -- 172.21.15.73:0/2676387320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 msgr2=0x55fa74c18e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74c18e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 -- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 msgr2=0x55fa74c17960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c17960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 -- 172.21.15.73:0/2676387320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fa74c17ea0 con 0x55fa74bf8080 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd582021640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74c18e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd581820640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd56c098330 tx=0x7fd56c086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5d113f983a88d1a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd565ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.126+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd56c09fa00 con 0x55fa74bf8080 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd56c09bbc0 con 0x55fa74bf8080 2024-03-09T00:08:50.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd5740058b0 con 0x55fa74bf8080 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd56c0b6e60 con 0x55fa74bf8080 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 10 client.16041 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fa74c788b0 con 0x55fa74bf8080 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5740058b0 con 0x55fa74bf8080 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd56c0a3c90 con 0x55fa74bf8080 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 10 client.16041 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 10 client.16041 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 20 client.16041 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:50.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c7af00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 4 client.16041 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fa74b68340 con 0x55fa74c21730 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.127+0000 7fd582021640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c7af00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd565ffb640 10 client.16041 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd582021640 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c7af00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab961c5569e6ae69 server_cookie=a5d15a880fda7a97 in_seq=0 out_seq=0 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd565ffb640 1 -- 172.21.15.73:0/2676387320 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fa74b68340 con 0x55fa74c21730 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd565ffb640 10 client.16041 handle_command_reply: tid=0 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd586e6f740 1 client.16041 shutdown 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 msgr2=0x55fa74c17420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.128+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fd56c098330 tx=0x7fd56c086040 comp rx=0 tx=0).stop 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 shutdown_connections 2024-03-09T00:08:50.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fa74c21730 0x55fa74c7af00 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fa74c00c60 0x55fa74c17960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fa74bf8080 0x55fa74c17420 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd565ffb640 0 client.16041 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 --2- 172.21.15.73:0/2676387320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fa74bf7760 0x55fa74c18e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 >> 172.21.15.73:0/2676387320 conn(0x55fa74bf1e90 msgr2=0x55fa74bf8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 shutdown_connections 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 1 -- 172.21.15.73:0/2676387320 wait complete. 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.129+0000 7fd586e6f740 20 client.16041 trim_cache size 0 max 16384 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:50.131 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:08:50.132 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:08:50.132 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:50.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 msgr2=0x7fd5680a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 0x7fd5680a4eb0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fd57802efd0 tx=0x7fd57802cf90 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 msgr2=0x7fd57c19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 crypto rx=0x7fd56c00e9f0 tx=0x7fd56c00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 shutdown_connections 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd57c10b930 0x7fd57c197a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5680a29f0 0x7fd5680a4eb0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd57c102e80 0x7fd57c19dd60 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 --2- 172.21.15.73:0/2431738651 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd57c102560 0x7fd57c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 >> 172.21.15.73:0/2431738651 conn(0x7fd57c077380 msgr2=0x7fd57c0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 shutdown_connections 2024-03-09T00:08:50.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.130+0000 7fd584285640 1 -- 172.21.15.73:0/2431738651 wait complete. 2024-03-09T00:08:50.142 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:08:50.142 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:08:50.149 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[2]: *** [include/buildrules:30: ltp] Error 2 2024-03-09T00:08:50.149 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[2]: *** Waiting for unfinished jobs.... 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 -- 172.21.15.73:0/40279748 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec101ed0 msgr2=0x7fb3ec105f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 --2- 172.21.15.73:0/40279748 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec101ed0 0x7fb3ec105f10 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8009900 tx=0x7fb3d802bc30 comp rx=0 tx=0).stop 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 -- 172.21.15.73:0/40279748 shutdown_connections 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 --2- 172.21.15.73:0/40279748 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec10e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 --2- 172.21.15.73:0/40279748 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec101ed0 0x7fb3ec105f10 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 --2- 172.21.15.73:0/40279748 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 -- 172.21.15.73:0/40279748 >> 172.21.15.73:0/40279748 conn(0x7fb3ec0fcd60 msgr2=0x7fb3ec0ff180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 -- 172.21.15.73:0/40279748 shutdown_connections 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.253+0000 7fb3f3252640 1 -- 172.21.15.73:0/40279748 wait complete. 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 Processor -- start 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 -- start start 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 0x7fb3ec197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec111680 con 0x7fb3ec1065f0 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec111500 con 0x7fb3ec101520 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec111800 con 0x7fb3ec197a40 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37936/0 (socket says 172.21.15.73:37936) 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 -- 172.21.15.73:0/354392453 learned_addr learned my addr 172.21.15.73:0/354392453 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f0fee640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f17ef640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 0x7fb3ec197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 -- 172.21.15.73:0/354392453 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 msgr2=0x7fb3ec197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 0x7fb3ec197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 -- 172.21.15.73:0/354392453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 msgr2=0x7fb3ec19d890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec19d890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 -- 172.21.15.73:0/354392453 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec198560 con 0x7fb3ec1065f0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f0fee640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f17ef640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 0x7fb3ec197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3ebfff640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8036040 tx=0x7fb3d802ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8041070 con 0x7fb3ec1065f0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.254+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec1987f0 con 0x7fb3ec1065f0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.255+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d80054e0 con 0x7fb3ec1065f0 2024-03-09T00:08:50.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.255+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8034440 con 0x7fb3ec1065f0 2024-03-09T00:08:50.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.255+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec1a4a40 con 0x7fb3ec1065f0 2024-03-09T00:08:50.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.256+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec0680e0 con 0x7fb3ec1065f0 2024-03-09T00:08:50.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.257+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb3d8044070 con 0x7fb3ec1065f0 2024-03-09T00:08:50.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.258+0000 7fb3e9ffb640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 0x7fb3dc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.258+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fb3d80f0f70 con 0x7fb3ec1065f0 2024-03-09T00:08:50.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.258+0000 7fb3f0fee640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 0x7fb3dc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.259+0000 7fb3f0fee640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 0x7fb3dc0a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0005d00 tx=0x7fb3e0005c70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.261+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fb3d80bdae0 con 0x7fb3ec1065f0 2024-03-09T00:08:50.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:50 smithi073 ceph-mon[32309]: pgmap v62: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.7 MiB/s wr, 1.02k op/s 2024-03-09T00:08:50.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.484+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3ec101ed0 con 0x7fb3ec1065f0 2024-03-09T00:08:50.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.484+0000 7fb3e9ffb640 1 -- 172.21.15.73:0/354392453 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fb3d8004c50 con 0x7fb3ec1065f0 2024-03-09T00:08:50.479 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:08:50.479 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:08:50.479 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:08:50.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 msgr2=0x7fb3dc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 0x7fb3dc0a4e30 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fb3e0005d00 tx=0x7fb3e0005c70 comp rx=0 tx=0).stop 2024-03-09T00:08:50.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 msgr2=0x7fb3ec19ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fb3d8036040 tx=0x7fb3d802ce20 comp rx=0 tx=0).stop 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 shutdown_connections 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb3ec197a40 0x7fb3ec197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3dc0a2970 0x7fb3dc0a4e30 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb3ec1065f0 0x7fb3ec19ddd0 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 --2- 172.21.15.73:0/354392453 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb3ec101520 0x7fb3ec19d890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.486+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 >> 172.21.15.73:0/354392453 conn(0x7fb3ec0fcd60 msgr2=0x7fb3ec0fe0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.487+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 shutdown_connections 2024-03-09T00:08:50.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.487+0000 7fb3f3252640 1 -- 172.21.15.73:0/354392453 wait complete. 2024-03-09T00:08:50.493 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:08:50.494 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:08:50.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:50 smithi102 ceph-mon[36106]: pgmap v62: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.7 MiB/s wr, 1.02k op/s 2024-03-09T00:08:50.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:50 smithi179 ceph-mon[36637]: pgmap v62: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 1021 KiB/s rd, 1.7 MiB/s wr, 1.02k op/s 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/2707313494 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb410b850 msgr2=0x7fcbb410dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/2707313494 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb410b850 0x7fcbb410dcf0 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7fcbac009ca0 tx=0x7fcbac02c180 comp rx=0 tx=0).stop 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/2707313494 shutdown_connections 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/2707313494 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb410b850 0x7fcbb410dcf0 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/2707313494 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 0x7fcbb410b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/2707313494 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb4102510 0x7fcbb41028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/2707313494 >> 172.21.15.73:0/2707313494 conn(0x7fcbb40773a0 msgr2=0x7fcbb40777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/2707313494 shutdown_connections 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/2707313494 wait complete. 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.646+0000 7fcbbbc4b640 1 Processor -- start 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 -- start start 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 0x7fcbb419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb410b850 0x7fcbb41978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4111610 con 0x7fcbb4102510 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4111490 con 0x7fcbb4102e30 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbbbc4b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb4111790 con 0x7fcbb410b850 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37966/0 (socket says 172.21.15.73:37966) 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3887941729 learned_addr learned my addr 172.21.15.73:0/3887941729 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 0x7fcbb419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb410b850 0x7fcbb41978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3887941729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb410b850 msgr2=0x7fcbb41978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb410b850 0x7fcbb41978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3887941729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 msgr2=0x7fcbb419dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 0x7fcbb419dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3887941729 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbb4198050 con 0x7fcbb4102510 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.647+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c920 tx=0x7fcbb000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.648+0000 7fcbaaffd640 1 -- 172.21.15.73:0/3887941729 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0007c70 con 0x7fcbb4102510 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.648+0000 7fcbaaffd640 1 -- 172.21.15.73:0/3887941729 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0007e10 con 0x7fcbb4102510 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.648+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbb4198340 con 0x7fcbb4102510 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.648+0000 7fcbaaffd640 1 -- 172.21.15.73:0/3887941729 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb0018890 con 0x7fcbb4102510 2024-03-09T00:08:50.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.648+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbb41a4590 con 0x7fcbb4102510 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.650+0000 7fcbbe82f740 1 Processor -- start 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.650+0000 7fcbaaffd640 1 -- 172.21.15.73:0/3887941729 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcbb0018a70 con 0x7fcbb4102510 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.650+0000 7fcbbe82f740 1 -- start start 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a1580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9aca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd968360 con 0x55accd9a1ac0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd904ec0 con 0x55accd9aa6a0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd8add30 con 0x55accd9a11a0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37976/0 (socket says 172.21.15.73:37976) 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 -- 172.21.15.73:0/1868136262 learned_addr learned my addr 172.21.15.73:0/1868136262 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbaaffd640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 0x7fcba00a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9aca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a1580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 -- 172.21.15.73:0/1868136262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 msgr2=0x55accd9a1580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a1580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 -- 172.21.15.73:0/1868136262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 msgr2=0x55accd9aca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9aca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 0x7fcba00a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 -- 172.21.15.73:0/1868136262 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55accd8acbc0 con 0x55accd9a1ac0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9aca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbaaffd640 1 -- 172.21.15.73:0/3887941729 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcbb00cf2b0 con 0x7fcbb4102510 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.651+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00cc5e0 tx=0x7fcbb0066240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a32de1efa64c75 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 0x7fcba00a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcbac006010 tx=0x7fcbac02e040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcb9ffff640 1 -- 172.21.15.73:0/1868136262 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb009aee0 con 0x55accd9a1ac0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcb9ffff640 1 -- 172.21.15.73:0/1868136262 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0066750 con 0x55accd9a1ac0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcb9ffff640 1 -- 172.21.15.73:0/1868136262 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00662b0 con 0x55accd9a1ac0 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1868136262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 msgr2=0x55accd9aa160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00cc5e0 tx=0x7fcbb0066240 comp rx=0 tx=0).stop 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1868136262 shutdown_connections 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9aca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9aa160 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1868136262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a1580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1868136262 >> 172.21.15.73:0/1868136262 conn(0x55accd99b8d0 msgr2=0x55accd99bcb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1868136262 shutdown_connections 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.652+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1868136262 wait complete. 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 -- start start 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 Processor -- start 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a76e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9a8160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd8a34a0 con 0x55accd9a1ac0 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd900f60 con 0x55accd9aa6a0 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55accd9ad3d0 con 0x55accd9a11a0 2024-03-09T00:08:50.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbbe82f740 10 client.0 _ll_register_callbacks cb 0x55accd8fed60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbb91e6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbb91e6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37982/0 (socket says 172.21.15.73:37982) 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbb91e6640 1 -- 172.21.15.73:0/1806197955 learned_addr learned my addr 172.21.15.73:0/1806197955 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.653+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a76e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9a8160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 -- 172.21.15.73:0/1806197955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 msgr2=0x55accd9a76e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a76e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 -- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 msgr2=0x55accd9a8160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9a8160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 -- 172.21.15.73:0/1806197955 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55accd6193f0 con 0x55accd9a1ac0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9a8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcba400b940 tx=0x7fcba400be00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=26f1781fd24582d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcba4005020 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcba40051c0 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcbb0066750 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcba4023050 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 10 client.16059 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55accd6e37d0 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb0066750 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcba4012610 con 0x55accd9a1ac0 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 10 client.16059 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 10 client.16059 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 20 client.16059 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.654+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accd9ba850 0x55accd9bcc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.655+0000 7fcbbe82f740 4 client.16059 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:50.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.655+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55accd912400 con 0x55accd9ba850 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.655+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accd9ba850 0x55accd9bcc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.655+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accd9ba850 0x55accd9bcc70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1749068b19755eb7 server_cookie=6eaac5cc60968c60 in_seq=0 out_seq=0 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.655+0000 7fcb9d7fa640 10 client.16059 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/1806197955 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55accd912400 con 0x55accd9ba850 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcb9d7fa640 10 client.16059 handle_command_reply: tid=0 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 client.16059 shutdown 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 msgr2=0x55accd9a7c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7fcba400b940 tx=0x7fcba400be00 comp rx=0 tx=0).stop 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 shutdown_connections 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accd9ba850 0x55accd9bcc70 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9aa6a0 0x55accd9a8160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a1ac0 0x55accd9a7c20 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/1806197955 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9a11a0 0x55accd9a76e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcb9d7fa640 0 client.16059 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 >> 172.21.15.73:0/1806197955 conn(0x55accd99b8d0 msgr2=0x55accd99d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 shutdown_connections 2024-03-09T00:08:50.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 1 -- 172.21.15.73:0/1806197955 wait complete. 2024-03-09T00:08:50.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.656+0000 7fcbbe82f740 20 client.16059 trim_cache size 0 max 16384 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 Processor -- start 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 -- start start 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9b6c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b7160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41ac6d0 con 0x55accd9a11a0 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41aec90 con 0x55accd9a1ac0 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41accd0 con 0x55accd9aa6a0 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:37992/0 (socket says 172.21.15.73:37992) 2024-03-09T00:08:50.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.668+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3133674367 learned_addr learned my addr 172.21.15.73:0/3133674367 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b7160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9b6c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3133674367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 msgr2=0x55accd9b7160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b7160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3133674367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 msgr2=0x55accd9b6c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9b6c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 -- 172.21.15.73:0/3133674367 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55accd9a18f0 con 0x55accd9a11a0 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbba1e8640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00cc720 tx=0x7fcbb006e040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8efe209ee029292e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcb9f7fe640 1 -- 172.21.15.73:0/3133674367 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb009aee0 con 0x55accd9a11a0 2024-03-09T00:08:50.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcb9f7fe640 1 -- 172.21.15.73:0/3133674367 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcbb00662b0 con 0x55accd9a11a0 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.669+0000 7fcb9f7fe640 1 -- 172.21.15.73:0/3133674367 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb00a3d20 con 0x55accd9a11a0 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- 172.21.15.73:0/3133674367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 msgr2=0x55accd9a9e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7fcbb00cc720 tx=0x7fcbb006e040 comp rx=0 tx=0).stop 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- 172.21.15.73:0/3133674367 shutdown_connections 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b7160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9b6c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/3133674367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9a9e10 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- 172.21.15.73:0/3133674367 >> 172.21.15.73:0/3133674367 conn(0x55accd99b8d0 msgr2=0x55accd9a5fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- 172.21.15.73:0/3133674367 shutdown_connections 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- 172.21.15.73:0/3133674367 wait complete. 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 -- start start 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.670+0000 7fcbbe82f740 1 Processor -- start 2024-03-09T00:08:50.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9afe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41ad310 con 0x55accd9a11a0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41ae630 con 0x55accd9a1ac0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbb41ad970 con 0x55accd9aa6a0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 10 client.0 _ll_register_callbacks cb 0x7fcbb00d2cd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38004/0 (socket says 172.21.15.73:38004) 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 -- 172.21.15.73:0/853135051 learned_addr learned my addr 172.21.15.73:0/853135051 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9afe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 -- 172.21.15.73:0/853135051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 msgr2=0x55accd9b0450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 -- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 msgr2=0x55accd9afe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9afe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 -- 172.21.15.73:0/853135051 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55accd9b0c50 con 0x55accd9a11a0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbb99e7640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0097260 tx=0x7fcbb009b1c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78b933752adfc48b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcb9d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcbb007a070 con 0x55accd9a11a0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcbb0099880 con 0x55accd9a11a0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.671+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55accd90c750 con 0x55accd9a11a0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcbb009bc80 con 0x55accd9a11a0 2024-03-09T00:08:50.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 10 client.16065 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55accda11ed0 con 0x55accd9a11a0 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55accd90c750 con 0x55accd9a11a0 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcbb00cc8f0 con 0x55accd9a11a0 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 10 client.16065 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 10 client.16065 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 20 client.16065 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accda12100 0x55accda144f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 4 client.16065 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55accd88a640 con 0x55accda12100 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.672+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accda12100 0x55accda144f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcb9d7fa640 10 client.16065 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbb91e6640 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accda12100 0x55accda144f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6c0d2571f5f7e008 server_cookie=99b421379b13ab65 in_seq=0 out_seq=0 2024-03-09T00:08:50.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcb9d7fa640 1 -- 172.21.15.73:0/853135051 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55accd88a640 con 0x55accda12100 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcb9d7fa640 10 client.16065 handle_command_reply: tid=0 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 client.16065 shutdown 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 msgr2=0x55accd9b1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7fcbb0097260 tx=0x7fcbb009b1c0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 shutdown_connections 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55accda12100 0x55accda144f0 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55accd9aa6a0 0x55accd9b0450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55accd9a1ac0 0x55accd9afe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcb9d7fa640 0 client.16065 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 --2- 172.21.15.73:0/853135051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55accd9a11a0 0x55accd9b1a80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 >> 172.21.15.73:0/853135051 conn(0x55accd99b8d0 msgr2=0x55accd9a5f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 shutdown_connections 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 1 -- 172.21.15.73:0/853135051 wait complete. 2024-03-09T00:08:50.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.673+0000 7fcbbe82f740 20 client.16065 trim_cache size 0 max 16384 2024-03-09T00:08:50.676 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 msgr2=0x7fcba00a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 0x7fcba00a4eb0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fcbac006010 tx=0x7fcbac02e040 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 msgr2=0x7fcbb419d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7fcbb000c920 tx=0x7fcbb000cde0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 shutdown_connections 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcbb410b850 0x7fcbb41978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcbb4102e30 0x7fcbb419dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcba00a29f0 0x7fcba00a4eb0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 --2- 172.21.15.73:0/3887941729 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcbb4102510 0x7fcbb419d6d0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 >> 172.21.15.73:0/3887941729 conn(0x7fcbb40773a0 msgr2=0x7fcbb40fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 shutdown_connections 2024-03-09T00:08:50.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:50.674+0000 7fcbbbc4b640 1 -- 172.21.15.73:0/3887941729 wait complete. 2024-03-09T00:08:50.790 INFO:tasks.workunit.client.0.smithi073.stdout: [DEP] 2024-03-09T00:08:50.790 INFO:tasks.workunit.client.0.smithi073.stdout: [DEP] 2024-03-09T00:08:50.793 INFO:tasks.workunit.client.0.smithi073.stdout: [DEP] 2024-03-09T00:08:51.011 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] dirstress 2024-03-09T00:08:51.013 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fill 2024-03-09T00:08:51.013 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fill2 2024-03-09T00:08:51.014 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] getpagesize 2024-03-09T00:08:51.146 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] holes 2024-03-09T00:08:51.189 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] lstat64 2024-03-09T00:08:51.246 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] nametest 2024-03-09T00:08:51.262 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] permname 2024-03-09T00:08:51.345 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] randholes 2024-03-09T00:08:51.363 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] runas 2024-03-09T00:08:51.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:51 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/354392453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:51.465 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] truncfile 2024-03-09T00:08:51.477 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] usemem 2024-03-09T00:08:51.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:51 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/354392453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:51.538 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] mmapcat 2024-03-09T00:08:51.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:51 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/354392453' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:08:51.611 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] append_reader 2024-03-09T00:08:51.628 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] append_writer 2024-03-09T00:08:51.647 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] dirperf 2024-03-09T00:08:51.680 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:08:51.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:08:51.681 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:08:51.699 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] metaperf 2024-03-09T00:08:51.761 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] devzero 2024-03-09T00:08:51.792 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] feature 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 -- 172.21.15.73:0/2486225163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c1065c0 msgr2=0x7fe06c10e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 --2- 172.21.15.73:0/2486225163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c1065c0 0x7fe06c10e190 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7fe064009ca0 tx=0x7fe06402c120 comp rx=0 tx=0).stop 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 -- 172.21.15.73:0/2486225163 shutdown_connections 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 --2- 172.21.15.73:0/2486225163 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c1065c0 0x7fe06c10e190 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 --2- 172.21.15.73:0/2486225163 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1003b0 0x7fe06c105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 --2- 172.21.15.73:0/2486225163 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c0ffa00 0x7fe06c0ffde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 -- 172.21.15.73:0/2486225163 >> 172.21.15.73:0/2486225163 conn(0x7fe06c076710 msgr2=0x7fe06c0fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 -- 172.21.15.73:0/2486225163 shutdown_connections 2024-03-09T00:08:51.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.821+0000 7fe071145640 1 -- 172.21.15.73:0/2486225163 wait complete. 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.822+0000 7fe071145640 1 Processor -- start 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.822+0000 7fe071145640 1 -- start start 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.822+0000 7fe071145640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe071145640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 0x7fe06c100eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe071145640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 0x7fe06c1013f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe071145640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c111660 con 0x7fe06c0ffa00 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe071145640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1114e0 con 0x7fe06c1065c0 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe071145640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1117e0 con 0x7fe06c102610 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38030/0 (socket says 172.21.15.73:38030) 2024-03-09T00:08:51.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 -- 172.21.15.73:0/1727397352 learned_addr learned my addr 172.21.15.73:0/1727397352 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06a59c640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 0x7fe06c100eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06b59e640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 0x7fe06c1013f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 -- 172.21.15.73:0/1727397352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 msgr2=0x7fe06c1013f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 0x7fe06c1013f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 -- 172.21.15.73:0/1727397352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 msgr2=0x7fe06c100eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 0x7fe06c100eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06ad9d640 1 -- 172.21.15.73:0/1727397352 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c1aa870 con 0x7fe06c0ffa00 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06a59c640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 0x7fe06c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.823+0000 7fe06b59e640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 0x7fe06c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.824+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.824+0000 7fe053fff640 1 -- 172.21.15.73:0/1727397352 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c00cd80 con 0x7fe06c0ffa00 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.824+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe06c1aab60 con 0x7fe06c0ffa00 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.824+0000 7fe053fff640 1 -- 172.21.15.73:0/1727397352 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe05c010910 con 0x7fe06c0ffa00 2024-03-09T00:08:51.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.824+0000 7fe053fff640 1 -- 172.21.15.73:0/1727397352 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c018990 con 0x7fe06c0ffa00 2024-03-09T00:08:51.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.825+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe06c1ab0b0 con 0x7fe06c0ffa00 2024-03-09T00:08:51.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.827+0000 7fe053fff640 1 -- 172.21.15.73:0/1727397352 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe05c010ac0 con 0x7fe06c0ffa00 2024-03-09T00:08:51.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 Processor -- start 2024-03-09T00:08:51.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe053fff640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 0x7fe0540a4e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 -- start start 2024-03-09T00:08:51.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe06b59e640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 0x7fe0540a4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e146d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe053fff640 1 -- 172.21.15.73:0/1727397352 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe05c0cf830 con 0x7fe06c0ffa00 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e149670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e0a1b40 con 0x55d93e13dd80 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e04ad80 con 0x55d93e147280 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.828+0000 7fe073e02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e09dbe0 con 0x55d93e13e6a0 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38046/0 (socket says 172.21.15.73:38046) 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2026564747 learned_addr learned my addr 172.21.15.73:0/2026564747 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06a59c640 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e146d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2026564747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 msgr2=0x55d93e146d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e146d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2026564747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 msgr2=0x55d93e149670 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-09T00:08:51.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e149670 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2026564747 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d93de7e7e0 con 0x55d93e13dd80 2024-03-09T00:08:51.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06b59e640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 0x7fe0540a4e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe06c102a80 tx=0x7fe06402c640 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0984d0 tx=0x7fe05c09c340 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa77b7df89b54127 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe0517fa640 1 -- 172.21.15.73:0/2026564747 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0b4070 con 0x55d93e13dd80 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe0517fa640 1 -- 172.21.15.73:0/2026564747 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe05c0a42a0 con 0x55d93e13dd80 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.829+0000 7fe0517fa640 1 -- 172.21.15.73:0/2026564747 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c018b30 con 0x55d93e13dd80 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 -- 172.21.15.73:0/2026564747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 msgr2=0x55d93e13e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fe05c0984d0 tx=0x7fe05c09c340 comp rx=0 tx=0).stop 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 -- 172.21.15.73:0/2026564747 shutdown_connections 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e146d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 --2- 172.21.15.73:0/2026564747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13dd80 0x55d93e13e160 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.830+0000 7fe073e02740 1 -- 172.21.15.73:0/2026564747 >> 172.21.15.73:0/2026564747 conn(0x55d93e1384b0 msgr2=0x55d93e138890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- 172.21.15.73:0/2026564747 shutdown_connections 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- 172.21.15.73:0/2026564747 wait complete. 2024-03-09T00:08:51.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- start start 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 Processor -- start 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e147280 0x55d93e144720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e1534e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e14a070 con 0x55d93e13e6a0 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e05e1c0 con 0x55d93e144c60 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.831+0000 7fe073e02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d93e14a1f0 con 0x55d93e147280 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe073e02740 10 client.0 _ll_register_callbacks cb 0x55d93e04d7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38060/0 (socket says 172.21.15.73:38060) 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2206660760 learned_addr learned my addr 172.21.15.73:0/2206660760 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2206660760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e147280 msgr2=0x55d93e144720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e147280 0x55d93e144720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 msgr2=0x55d93e1534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e1534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 -- 172.21.15.73:0/2206660760 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d93dedef70 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe05c09ba40 tx=0x7fe05c018420 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c7b5eb87d4e140 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe05c0b4040 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe05c099e00 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe05c0a42a0 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.832+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe05c0ba220 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 10 client.16077 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d93dd76170 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe05c0a42a0 con 0x55d93e13e6a0 2024-03-09T00:08:51.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe05c064070 con 0x55d93e13e6a0 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 10 client.16077 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 10 client.16077 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 20 client.16077 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e157830 0x55d93e159c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 4 client.16077 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d93e077960 con 0x55d93e157830 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.833+0000 7fe06a59c640 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e157830 0x55d93e159c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.834+0000 7fe06a59c640 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e157830 0x55d93e159c20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=da5d94b502290bcc server_cookie=55b04d707c985283 in_seq=0 out_seq=0 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.834+0000 7fe04eff5640 10 client.16077 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.834+0000 7fe04eff5640 1 -- 172.21.15.73:0/2206660760 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d93e077960 con 0x55d93e157830 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.834+0000 7fe04eff5640 10 client.16077 handle_command_reply: tid=0 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.834+0000 7fe073e02740 1 client.16077 shutdown 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 msgr2=0x55d93e1441e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fe05c09ba40 tx=0x7fe05c018420 comp rx=0 tx=0).stop 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 shutdown_connections 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e157830 0x55d93e159c20 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e1534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e147280 0x55d93e144720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 --2- 172.21.15.73:0/2206660760 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e13e6a0 0x55d93e1441e0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 >> 172.21.15.73:0/2206660760 conn(0x55d93e1384b0 msgr2=0x55d93e13a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe04eff5640 0 client.16077 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 shutdown_connections 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 1 -- 172.21.15.73:0/2206660760 wait complete. 2024-03-09T00:08:51.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.835+0000 7fe073e02740 20 client.16077 trim_cache size 0 max 16384 2024-03-09T00:08:51.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.856+0000 7fe073e02740 1 Processor -- start 2024-03-09T00:08:51.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.856+0000 7fe073e02740 1 -- start start 2024-03-09T00:08:51.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14e430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e14e970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b2e90 con 0x55d93e147280 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b5450 con 0x55d93e144c60 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe073e02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b3490 con 0x55d93e13e6a0 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38070/0 (socket says 172.21.15.73:38070) 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 -- 172.21.15.73:0/4249620326 learned_addr learned my addr 172.21.15.73:0/4249620326 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06a59c640 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e14e970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14e430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 -- 172.21.15.73:0/4249620326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 msgr2=0x55d93e14e430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14e430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 -- 172.21.15.73:0/4249620326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 msgr2=0x55d93e14e970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e14e970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.857+0000 7fe06b59e640 1 -- 172.21.15.73:0/4249620326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d93e1499e0 con 0x55d93e147280 2024-03-09T00:08:51.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.858+0000 7fe06b59e640 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe06402c730 tx=0x7fe06402e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b4a2e5159267c4c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.858+0000 7fe050ff9640 1 -- 172.21.15.73:0/4249620326 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064050070 con 0x55d93e147280 2024-03-09T00:08:51.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.858+0000 7fe050ff9640 1 -- 172.21.15.73:0/4249620326 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe064047040 con 0x55d93e147280 2024-03-09T00:08:51.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.858+0000 7fe050ff9640 1 -- 172.21.15.73:0/4249620326 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064046050 con 0x55d93e147280 2024-03-09T00:08:51.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- 172.21.15.73:0/4249620326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 msgr2=0x55d93e142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7fe06402c730 tx=0x7fe06402e050 comp rx=0 tx=0).stop 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- 172.21.15.73:0/4249620326 shutdown_connections 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e147280 0x55d93e142ab0 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e144c60 0x55d93e14e970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 --2- 172.21.15.73:0/4249620326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14e430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- 172.21.15.73:0/4249620326 >> 172.21.15.73:0/4249620326 conn(0x55d93e1384b0 msgr2=0x55d93e152ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- 172.21.15.73:0/4249620326 shutdown_connections 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- 172.21.15.73:0/4249620326 wait complete. 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 -- start start 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.859+0000 7fe073e02740 1 Processor -- start 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e1ada00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b3ad0 con 0x55d93e144c60 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b4df0 con 0x55d93e147280 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe06c1b4130 con 0x55d93e13e6a0 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe073e02740 10 client.0 _ll_register_callbacks cb 0x7fe05c097680 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe06a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38076/0 (socket says 172.21.15.73:38076) 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe06a59c640 1 -- 172.21.15.73:0/691776632 learned_addr learned my addr 172.21.15.73:0/691776632 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.860+0000 7fe06b59e640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e1ada00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 -- 172.21.15.73:0/691776632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 msgr2=0x55d93e14d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 -- 172.21.15.73:0/691776632 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 msgr2=0x55d93e14d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 -- 172.21.15.73:0/691776632 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d93e14df40 con 0x55d93e147280 2024-03-09T00:08:51.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06b59e640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e1ada00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe064009bf0 tx=0x7fe064004970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ae5974d62cc1acc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe04eff5640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06a59c640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe064059020 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe064047040 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe064003970 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe064046020 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe073e02740 10 client.25490 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.861+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d93e1af080 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe064047040 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe06404bb60 con 0x55d93e147280 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 10 client.25490 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:51.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 10 client.25490 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 20 client.25490 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e1af760 0x55d93e1b1b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 4 client.25490 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55d93e063080 con 0x55d93e1af760 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.862+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e1af760 0x55d93e1b1b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.863+0000 7fe04eff5640 10 client.25490 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:51.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.863+0000 7fe06ad9d640 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e1af760 0x55d93e1b1b80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9ac7f20f0eac0fa5 server_cookie=641343e6c9524306 in_seq=0 out_seq=0 2024-03-09T00:08:51.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe04eff5640 1 -- 172.21.15.73:0/691776632 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55d93e063080 con 0x55d93e1af760 2024-03-09T00:08:51.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe04eff5640 10 client.25490 handle_command_reply: tid=0 2024-03-09T00:08:51.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 client.25490 shutdown 2024-03-09T00:08:51.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 msgr2=0x55d93e1ada00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e1ada00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe064009bf0 tx=0x7fe064004970 comp rx=0 tx=0).stop 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 shutdown_connections 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d93e1af760 0x55d93e1b1b80 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d93e147280 0x55d93e1ada00 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe04eff5640 0 client.25490 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d93e144c60 0x55d93e14d650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 --2- 172.21.15.73:0/691776632 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d93e13e6a0 0x55d93e14d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 >> 172.21.15.73:0/691776632 conn(0x55d93e1384b0 msgr2=0x55d93e142a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 shutdown_connections 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.864+0000 7fe073e02740 1 -- 172.21.15.73:0/691776632 wait complete. 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe073e02740 20 client.25490 trim_cache size 0 max 16384 2024-03-09T00:08:51.863 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:51.864 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:08:51.864 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "b94e8871-709b-420c-b0d5-7076e596a23c", 2024-03-09T00:08:51.864 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:08:51.864 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 msgr2=0x7fe0540a4e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 0x7fe0540a4e70 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7fe06c102a80 tx=0x7fe06402c640 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 msgr2=0x7fe06c100970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7fe05c00e9f0 tx=0x7fe05c00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 shutdown_connections 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe06c102610 0x7fe06c1013f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe06c1065c0 0x7fe06c100eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0540a29b0 0x7fe0540a4e70 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 --2- 172.21.15.73:0/1727397352 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe06c0ffa00 0x7fe06c100970 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:51.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 >> 172.21.15.73:0/1727397352 conn(0x7fe06c076710 msgr2=0x7fe06c1048b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:51.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 shutdown_connections 2024-03-09T00:08:51.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:51.865+0000 7fe071145640 1 -- 172.21.15.73:0/1727397352 wait complete. 2024-03-09T00:08:51.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:08:51.879 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:08:51.900 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] alloc 2024-03-09T00:08:51.967 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fault 2024-03-09T00:08:52.058 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] fstest 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f87577fe640 1 -- 172.21.15.73:0/3685290725 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874c02cc80 con 0x7f8758100d80 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 -- 172.21.15.73:0/3685290725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758100d80 msgr2=0x7f8758104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 --2- 172.21.15.73:0/3685290725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758100d80 0x7f8758104dc0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f874c009900 tx=0x7f874c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 -- 172.21.15.73:0/3685290725 shutdown_connections 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 --2- 172.21.15.73:0/3685290725 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581054a0 0x7f875810d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 --2- 172.21.15.73:0/3685290725 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8758100d80 0x7f8758104dc0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 --2- 172.21.15.73:0/3685290725 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581003d0 0x7f87581007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.051+0000 7f875f034640 1 -- 172.21.15.73:0/3685290725 >> 172.21.15.73:0/3685290725 conn(0x7f87580772f0 msgr2=0x7f8758077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.052+0000 7f875f034640 1 -- 172.21.15.73:0/3685290725 shutdown_connections 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.052+0000 7f875f034640 1 -- 172.21.15.73:0/3685290725 wait complete. 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.052+0000 7f875f034640 1 Processor -- start 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.052+0000 7f875f034640 1 -- start start 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581054a0 0x7f8758196d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 0x7f875819b810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758110580 con 0x7f87581972e0 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758110400 con 0x7f87581003d0 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875f034640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758110700 con 0x7f87581054a0 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:51460/0 (socket says 172.21.15.73:51460) 2024-03-09T00:08:52.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 -- 172.21.15.73:0/2817311213 learned_addr learned my addr 172.21.15.73:0/2817311213 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f8757fff640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581054a0 0x7f8758196d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 0x7f875819b810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 -- 172.21.15.73:0/2817311213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581054a0 msgr2=0x7f8758196d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581054a0 0x7f8758196d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 -- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 msgr2=0x7f875819b810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 0x7f875819b810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875cdd0640 1 -- 172.21.15.73:0/2817311213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f875819c0a0 con 0x7f87581003d0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.053+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 0x7f875819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f875cdd0640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f874800e9f0 tx=0x7f874800edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f8755ffb640 1 -- 172.21.15.73:0/2817311213 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874800cd80 con 0x7f87581003d0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f8755ffb640 1 -- 172.21.15.73:0/2817311213 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8748005500 con 0x7f87581003d0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f8755ffb640 1 -- 172.21.15.73:0/2817311213 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8748010690 con 0x7f87581003d0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f875819c390 con 0x7f87581003d0 2024-03-09T00:08:52.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.054+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f875819c860 con 0x7f87581003d0 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.057+0000 7f8761c0d740 1 Processor -- start 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 -- start start 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8755ffb640 1 -- 172.21.15.73:0/2817311213 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f87480046c0 con 0x7f87581003d0 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07f050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5afd6e00 con 0x55ed5b073760 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f875d5d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5af7fb10 con 0x55ed5b074080 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f8761c0d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5afd2ea0 con 0x55ed5b07cc60 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f875d5d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38114/0 (socket says 172.21.15.73:38114) 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f875d5d1640 1 -- 172.21.15.73:0/2434369227 learned_addr learned my addr 172.21.15.73:0/2434369227 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.058+0000 7f875cdd0640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f8757fff640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f875d5d1640 1 -- 172.21.15.73:0/2434369227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 msgr2=0x55ed5b07f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f8755ffb640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 0x7f87400a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.066 INFO:tasks.workunit.client.0.smithi073.stdout: [CC] t_access_root 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f875d5d1640 1 -- 172.21.15.73:0/2434369227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 msgr2=0x55ed5b07c720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.059+0000 7f875d5d1640 1 -- 172.21.15.73:0/2434369227 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed5adb57d0 con 0x55ed5b073760 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f875cdd0640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f8755ffb640 1 -- 172.21.15.73:0/2817311213 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f87480cf5e0 con 0x7f87581003d0 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f87500107a0 tx=0x7f8750010c60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a8bc7c923a3d7279 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f873effd640 1 -- 172.21.15.73:0/2434369227 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875000eb30 con 0x55ed5b073760 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 0x7f87400a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f873effd640 1 -- 172.21.15.73:0/2434369227 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8750010e50 con 0x55ed5b073760 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.060+0000 7f873effd640 1 -- 172.21.15.73:0/2434369227 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8750019610 con 0x55ed5b073760 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f875d5d1640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 0x7f87400a4f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f875001a000 tx=0x7f87500199c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 -- 172.21.15.73:0/2434369227 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 msgr2=0x55ed5b073b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 crypto rx=0x7f87500107a0 tx=0x7f8750010c60 comp rx=0 tx=0).stop 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 -- 172.21.15.73:0/2434369227 shutdown_connections 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 --2- 172.21.15.73:0/2434369227 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b073b40 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.061+0000 7f8761c0d740 1 -- 172.21.15.73:0/2434369227 >> 172.21.15.73:0/2434369227 conn(0x55ed5b06de90 msgr2=0x55ed5b06e270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.062+0000 7f8761c0d740 1 -- 172.21.15.73:0/2434369227 shutdown_connections 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.062+0000 7f8761c0d740 1 -- 172.21.15.73:0/2434369227 wait complete. 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.062+0000 7f8761c0d740 1 -- start start 2024-03-09T00:08:52.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.062+0000 7f8761c0d740 1 Processor -- start 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b073760 0x55ed5b07c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 0x55ed5b079600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5b07faf0 con 0x55ed5b07cc60 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5af930a0 con 0x55ed5b073760 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed5b07fc70 con 0x55ed5b074080 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8761c0d740 10 client.0 _ll_register_callbacks cb 0x55ed5af92a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f8757fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 0x55ed5b079600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f875cdd0640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b073760 0x55ed5b07c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.063+0000 7f875d5d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38124/0 (socket says 172.21.15.73:38124) 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 -- 172.21.15.73:0/1139314112 learned_addr learned my addr 172.21.15.73:0/1139314112 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 -- 172.21.15.73:0/1139314112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 msgr2=0x55ed5b079600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 0x55ed5b079600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 -- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b073760 msgr2=0x55ed5b07c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.064+0000 7f875d5d1640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b073760 0x55ed5b07c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f875d5d1640 1 -- 172.21.15.73:0/1139314112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed5aceb3f0 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8757fff640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 0x55ed5b079600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f875d5d1640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8750011fc0 tx=0x7f8750023000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d41f74e5e9ece68f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8737fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f875001f040 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f87500049c0 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8750010e50 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8750004bc0 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8761c0d740 10 client.16095 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.065+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ed5acab0f0 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.066+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8750010e50 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.066+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8750030070 con 0x55ed5b07cc60 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.066+0000 7f8761c0d740 10 client.16095 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:52.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8761c0d740 10 client.16095 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:52.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8761c0d740 20 client.16095 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:52.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b09d650 0x55ed5b09fa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8761c0d740 4 client.16095 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:08:52.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed5afe4340 con 0x55ed5b09d650 2024-03-09T00:08:52.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f875cdd0640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b09d650 0x55ed5b09fa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.073 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c: In function 'main': 2024-03-09T00:08:52.073 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:154:1: error: 'bozo' undeclared (first use in this function) 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr: 154 | bozo! 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~ 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:154:1: note: each undeclared identifier is reported only once for each function it appears in 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:154:5: error: expected ';' before '!' token 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr: 154 | bozo! 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr: | ^ 2024-03-09T00:08:52.074 INFO:tasks.workunit.client.0.smithi073.stderr: | ; 2024-03-09T00:08:52.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f8737fff640 10 client.16095 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:52.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.067+0000 7f875cdd0640 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b09d650 0x55ed5b09fa40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3051dd0bdbf042d server_cookie=703e263bbf897cb9 in_seq=0 out_seq=0 2024-03-09T00:08:52.075 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:184:25: error: 'rflag' undeclared (first use in this function); did you mean 'nflag'? 2024-03-09T00:08:52.075 INFO:tasks.workunit.client.0.smithi073.stderr: 184 | rflag = 1; 2024-03-09T00:08:52.075 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~ 2024-03-09T00:08:52.075 INFO:tasks.workunit.client.0.smithi073.stderr: | nflag 2024-03-09T00:08:52.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.068+0000 7f8737fff640 1 -- 172.21.15.73:0/1139314112 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ed5afe4340 con 0x55ed5b09d650 2024-03-09T00:08:52.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.068+0000 7f8737fff640 10 client.16095 handle_command_reply: tid=0 2024-03-09T00:08:52.076 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:320:5: error: expected ';' before '!' token 2024-03-09T00:08:52.076 INFO:tasks.workunit.client.0.smithi073.stderr: 320 | bozo! 2024-03-09T00:08:52.076 INFO:tasks.workunit.client.0.smithi073.stderr: | ^ 2024-03-09T00:08:52.076 INFO:tasks.workunit.client.0.smithi073.stderr: | ; 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.068+0000 7f8761c0d740 1 client.16095 shutdown 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 msgr2=0x55ed5b079b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 crypto rx=0x7f8750011fc0 tx=0x7f8750023000 comp rx=0 tx=0).stop 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 shutdown_connections 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b09d650 0x55ed5b09fa40 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b07cc60 0x55ed5b079b40 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b074080 0x55ed5b079600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 --2- 172.21.15.73:0/1139314112 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b073760 0x55ed5b07c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 >> 172.21.15.73:0/1139314112 conn(0x55ed5b06de90 msgr2=0x55ed5b06fbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8737fff640 0 client.16095 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 shutdown_connections 2024-03-09T00:08:52.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 1 -- 172.21.15.73:0/1139314112 wait complete. 2024-03-09T00:08:52.078 INFO:tasks.workunit.client.0.smithi073.stderr:alloc.c:128:25: warning: variable 'f' set but not used [-Wunused-but-set-variable] 2024-03-09T00:08:52.078 INFO:tasks.workunit.client.0.smithi073.stderr: 128 | struct flock64 f; 2024-03-09T00:08:52.078 INFO:tasks.workunit.client.0.smithi073.stderr: | ^ 2024-03-09T00:08:52.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.069+0000 7f8761c0d740 20 client.16095 trim_cache size 0 max 16384 2024-03-09T00:08:52.078 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[3]: *** [Makefile:83: alloc] Error 1 2024-03-09T00:08:52.078 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[3]: *** Waiting for unfinished jobs.... 2024-03-09T00:08:52.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 Processor -- start 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 -- start start 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b07bd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87580724f0 con 0x55ed5b073760 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87581acbc0 con 0x55ed5b074080 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.093+0000 7f8761c0d740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758072af0 con 0x55ed5b07cc60 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f8757fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b07bd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f8757fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:51490/0 (socket says 172.21.15.73:51490) 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f8757fff640 1 -- 172.21.15.73:0/216109144 learned_addr learned my addr 172.21.15.73:0/216109144 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875d5d1640 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 -- 172.21.15.73:0/216109144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 msgr2=0x55ed5b07e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 -- 172.21.15.73:0/216109144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 msgr2=0x55ed5b07c270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 -- 172.21.15.73:0/216109144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed5b07f930 con 0x55ed5b073760 2024-03-09T00:08:52.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f8757fff640 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f875cdd0640 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b07bd30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f87480cc940 tx=0x7f8748086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cc131c6545114603 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.094+0000 7f873e7fc640 1 -- 172.21.15.73:0/216109144 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874809b390 con 0x55ed5b073760 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f873e7fc640 1 -- 172.21.15.73:0/216109144 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f874809bf50 con 0x55ed5b073760 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f873e7fc640 1 -- 172.21.15.73:0/216109144 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f874809b1b0 con 0x55ed5b073760 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 -- 172.21.15.73:0/216109144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 msgr2=0x55ed5b07bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b07bd30 secure :-1 s=READY pgs=653 cs=0 l=1 rev1=1 crypto rx=0x7f87480cc940 tx=0x7f8748086040 comp rx=0 tx=0).stop 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 -- 172.21.15.73:0/216109144 shutdown_connections 2024-03-09T00:08:52.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b07cc60 0x55ed5b07e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b074080 0x55ed5b07c270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.095+0000 7f8761c0d740 1 --2- 172.21.15.73:0/216109144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b073760 0x55ed5b07bd30 unknown :-1 s=CLOSED pgs=653 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 -- 172.21.15.73:0/216109144 >> 172.21.15.73:0/216109144 conn(0x55ed5b06de90 msgr2=0x55ed5b0795c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 -- 172.21.15.73:0/216109144 shutdown_connections 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 -- 172.21.15.73:0/216109144 wait complete. 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 -- start start 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 Processor -- start 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.096+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 0x55ed5b083eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8757fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8757fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:38136/0 (socket says 172.21.15.73:38136) 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8757fff640 1 -- 172.21.15.73:0/4016228625 learned_addr learned my addr 172.21.15.73:0/4016228625 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:08:52.094 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b07cc60 0x55ed5b075de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8758073130 con 0x55ed5b074080 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87581ac560 con 0x55ed5b07cc60 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f875cdd0640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 0x55ed5b083eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87581ab8a0 con 0x55ed5b073760 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f8761c0d740 10 client.0 _ll_register_callbacks cb 0x7f87500105f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.097+0000 7f875d5d1640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b07cc60 0x55ed5b075de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 -- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b07cc60 msgr2=0x55ed5b075de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b07cc60 0x55ed5b075de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 -- 172.21.15.73:0/4016228625 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 msgr2=0x55ed5b0843f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed5b0765e0 con 0x55ed5b073760 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f8757fff640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f875cdd0640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 0x55ed5b083eb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2c80 tx=0x7f874809b6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=644b92b929eec399 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:08:52.095 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f8737fff640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f87480b3070 con 0x55ed5b073760 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f874809bf50 con 0x55ed5b073760 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.098+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f87480ccbc0 con 0x55ed5b073760 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.099+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f874809b850 con 0x55ed5b073760 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.099+0000 7f8761c0d740 10 client.25576 fetch_fsmap learned FSMap version 20 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.099+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ed5b0f44e0 con 0x55ed5b073760 2024-03-09T00:08:52.096 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.099+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f874809bf50 con 0x55ed5b073760 2024-03-09T00:08:52.097 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f87480a7070 con 0x55ed5b073760 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 10 client.25576 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 10 client.25576 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 20 client.25576 populate_metadata read hostname 'smithi073' 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b0f4ab0 0x55ed5b0f6ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 4 client.25576 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ed5afde690 con 0x55ed5b0f4ab0 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8757fff640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b0f4ab0 0x55ed5b0f6ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.100+0000 7f8757fff640 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b0f4ab0 0x55ed5b0f6ed0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5e7d337961897524 server_cookie=c1dd96106ddae7f8 in_seq=0 out_seq=0 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8737fff640 10 client.25576 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8737fff640 1 -- 172.21.15.73:0/4016228625 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55ed5afde690 con 0x55ed5b0f4ab0 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8737fff640 10 client.25576 handle_command_reply: tid=0 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 client.25576 shutdown 2024-03-09T00:08:52.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 msgr2=0x55ed5b083eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 0x55ed5b083eb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f87480d2c80 tx=0x7f874809b6e0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 shutdown_connections 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ed5b0f4ab0 0x55ed5b0f6ed0 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ed5b07cc60 0x55ed5b075de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ed5b074080 0x55ed5b0843f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 --2- 172.21.15.73:0/4016228625 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ed5b073760 0x55ed5b083eb0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 >> 172.21.15.73:0/4016228625 conn(0x55ed5b06de90 msgr2=0x55ed5b074460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8737fff640 0 client.25576 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 shutdown_connections 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.101+0000 7f8761c0d740 1 -- 172.21.15.73:0/4016228625 wait complete. 2024-03-09T00:08:52.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.102+0000 7f8761c0d740 20 client.25576 trim_cache size 0 max 16384 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (1802 inodes in the stack)", 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "b94e8871-709b-420c-b0d5-7076e596a23c": { 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "b94e8871-709b-420c-b0d5-7076e596a23c", 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:08:52.100 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 msgr2=0x7f87400a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 0x7f87400a4f30 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f875001a000 tx=0x7f87500199c0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 msgr2=0x7f87581967c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f874800e9f0 tx=0x7f874800edf0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 shutdown_connections 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f87400a2a70 0x7f87400a4f30 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f87581972e0 0x7f875819b810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f87581054a0 0x7f8758196d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 --2- 172.21.15.73:0/2817311213 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f87581003d0 0x7f87581967c0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:08:52.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 >> 172.21.15.73:0/2817311213 conn(0x7f87580772f0 msgr2=0x7f87580fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:08:52.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 shutdown_connections 2024-03-09T00:08:52.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:08:52.103+0000 7f875f034640 1 -- 172.21.15.73:0/2817311213 wait complete. 2024-03-09T00:08:52.115 INFO:tasks.cephfs.filesystem:scrub status for tag:b94e8871-709b-420c-b0d5-7076e596a23c - {'path': '/', 'tag': 'b94e8871-709b-420c-b0d5-7076e596a23c', 'options': 'recursive,force'} 2024-03-09T00:08:52.229 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c: In function 'run_child': 2024-03-09T00:08:52.230 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:95:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-09T00:08:52.230 INFO:tasks.workunit.client.0.smithi073.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.230 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~ 2024-03-09T00:08:52.230 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:95:24: note: directive argument in the range [0, 2147483647] 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~ 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:95:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr: 95 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:166:27: warning: '/file' directive writing 5 bytes into a region of size between 1 and 1024 [-Wformat-overflow=] 2024-03-09T00:08:52.231 INFO:tasks.workunit.client.0.smithi073.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.232 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~ 2024-03-09T00:08:52.232 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:166:24: note: directive argument in the range [0, 2147483647] 2024-03-09T00:08:52.232 INFO:tasks.workunit.client.0.smithi073.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.232 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~ 2024-03-09T00:08:52.232 INFO:tasks.workunit.client.0.smithi073.stderr:fstest.c:166:9: note: 'sprintf' output between 7 and 1039 bytes into a destination of size 1024 2024-03-09T00:08:52.233 INFO:tasks.workunit.client.0.smithi073.stderr: 166 | sprintf(fname, "%s/file%d", dir, fnum); 2024-03-09T00:08:52.233 INFO:tasks.workunit.client.0.smithi073.stderr: | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-03-09T00:08:52.304 INFO:tasks.workunit.client.0.smithi073.stderr:gmake[2]: *** [include/buildrules:30: src] Error 2 2024-03-09T00:08:52.304 INFO:tasks.workunit.client.0.smithi073.stderr:make[1]: *** [Makefile:53: default] Error 2 2024-03-09T00:08:52.304 INFO:tasks.workunit.client.0.smithi073.stderr:make: *** [Makefile:51: default] Error 2 2024-03-09T00:08:52.306 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-09T00:08:52.307 INFO:tasks.workunit:Stopping ['suites/fsx.sh'] on client.0... 2024-03-09T00:08:52.307 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-09T00:08:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:52 smithi073 ceph-mon[32309]: pgmap v63: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 900 KiB/s wr, 742 op/s 2024-03-09T00:08:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:52 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:08:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:52 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:52 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:08:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:52 smithi102 ceph-mon[36106]: pgmap v63: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 900 KiB/s wr, 742 op/s 2024-03-09T00:08:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:52 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:08:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:52 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:52 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:08:52.572 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi073 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-09T00:08:52.883 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=bc7102cb35984a438080376b4db19bbe Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_7d63482aa89628d733e298984a762389b625c761/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/fsx.sh) on smithi073 with status 2: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh' 2024-03-09T00:08:52.886 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-09T00:08:52.961 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:08:52.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:52 smithi179 ceph-mon[36637]: pgmap v63: 129 pgs: 129 active+clean; 24 MiB data, 619 MiB used, 1.0 TiB / 1.0 TiB avail; 963 KiB/s rd, 900 KiB/s wr, 742 op/s 2024-03-09T00:08:52.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:52 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:08:52.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:52 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:52.967 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:52 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:08:52.979 DEBUG:teuthology.run_tasks:Unwinding manager full_sequential_finally 2024-03-09T00:08:53.698 INFO:teuthology.task.full_sequential_finally:In full_sequential_finally, running task exec... 2024-03-09T00:08:53.698 INFO:teuthology.task.exec:Executing custom commands... 2024-03-09T00:08:53.699 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:08:53.699 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'date +%s > END_TIME' 2024-03-09T00:08:53.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:53 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:53.701 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:53 smithi179 ceph-mon[36637]: pgmap v64: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 804 op/s 2024-03-09T00:08:53.701 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:53 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:53.701 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:53 smithi102 ceph-mon[36106]: pgmap v64: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 804 op/s 2024-03-09T00:08:53.702 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:53 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:08:53.702 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:53 smithi073 ceph-mon[32309]: pgmap v64: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 994 KiB/s wr, 804 op/s 2024-03-09T00:08:53.735 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'START_TIME=$(cat START_TIME); END_TIME=$(cat END_TIME); DIFF_TIME=$((600-(END_TIME-START_TIME))); if [ "$DIFF_TIME" -gt 0 ]; then sleep "$DIFF_TIME"; fi' 2024-03-09T00:08:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:54 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:08:54.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:54 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:08:54.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:54 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:08:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:55 smithi073 ceph-mon[32309]: pgmap v65: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 317 KiB/s wr, 99 op/s 2024-03-09T00:08:55.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:55 smithi102 ceph-mon[36106]: pgmap v65: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 317 KiB/s wr, 99 op/s 2024-03-09T00:08:55.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:55 smithi179 ceph-mon[36637]: pgmap v65: 129 pgs: 129 active+clean; 25 MiB data, 607 MiB used, 1.0 TiB / 1.0 TiB avail; 373 KiB/s rd, 317 KiB/s wr, 99 op/s 2024-03-09T00:08:58.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:58 smithi179 ceph-mon[36637]: pgmap v66: 129 pgs: 129 active+clean; 28 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 644 KiB/s wr, 265 op/s 2024-03-09T00:08:58.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:58 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:08:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:58 smithi073 ceph-mon[32309]: pgmap v66: 129 pgs: 129 active+clean; 28 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 644 KiB/s wr, 265 op/s 2024-03-09T00:08:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:58 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:08:58.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:58 smithi102 ceph-mon[36106]: pgmap v66: 129 pgs: 129 active+clean; 28 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 496 KiB/s rd, 644 KiB/s wr, 265 op/s 2024-03-09T00:08:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:58 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:08:59.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:08:59.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:08:59.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:08:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:08:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:08:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:08:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:08:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:08:59.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:08:59.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:08:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:08:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:09:00.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:00 smithi179 ceph-mon[36637]: pgmap v67: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 810 KiB/s wr, 390 op/s 2024-03-09T00:09:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:00 smithi073 ceph-mon[32309]: pgmap v67: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 810 KiB/s wr, 390 op/s 2024-03-09T00:09:00.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:00 smithi102 ceph-mon[36106]: pgmap v67: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 612 KiB/s rd, 810 KiB/s wr, 390 op/s 2024-03-09T00:09:02.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:02 smithi179 ceph-mon[36637]: pgmap v68: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 587 KiB/s wr, 368 op/s 2024-03-09T00:09:02.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:02 smithi073 ceph-mon[32309]: pgmap v68: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 587 KiB/s wr, 368 op/s 2024-03-09T00:09:02.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:02 smithi102 ceph-mon[36106]: pgmap v68: 129 pgs: 129 active+clean; 30 MiB data, 626 MiB used, 1.0 TiB / 1.0 TiB avail; 418 KiB/s rd, 587 KiB/s wr, 368 op/s 2024-03-09T00:09:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:04 smithi179 ceph-mon[36637]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.2 MiB/s wr, 455 op/s 2024-03-09T00:09:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:04 smithi073 ceph-mon[32309]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.2 MiB/s wr, 455 op/s 2024-03-09T00:09:04.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:04 smithi102 ceph-mon[36106]: pgmap v69: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 678 KiB/s rd, 1.2 MiB/s wr, 455 op/s 2024-03-09T00:09:06.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:06 smithi179 ceph-mon[36637]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-09T00:09:06.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:06 smithi073 ceph-mon[32309]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-09T00:09:06.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:06 smithi102 ceph-mon[36106]: pgmap v70: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 499 KiB/s rd, 1.1 MiB/s wr, 393 op/s 2024-03-09T00:09:08.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:08 smithi179 ceph-mon[36637]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-09T00:09:08.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:08 smithi073 ceph-mon[32309]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-09T00:09:08.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:08 smithi102 ceph-mon[36106]: pgmap v71: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 504 KiB/s rd, 1.1 MiB/s wr, 395 op/s 2024-03-09T00:09:10.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:10 smithi179 ceph-mon[36637]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 789 KiB/s wr, 230 op/s 2024-03-09T00:09:10.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:10 smithi073 ceph-mon[32309]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 789 KiB/s wr, 230 op/s 2024-03-09T00:09:10.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:10 smithi102 ceph-mon[36106]: pgmap v72: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 381 KiB/s rd, 789 KiB/s wr, 230 op/s 2024-03-09T00:09:12.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:12 smithi179 ceph-mon[36637]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 622 KiB/s wr, 89 op/s 2024-03-09T00:09:12.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:12 smithi073 ceph-mon[32309]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 622 KiB/s wr, 89 op/s 2024-03-09T00:09:12.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:12 smithi102 ceph-mon[36106]: pgmap v73: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 265 KiB/s rd, 622 KiB/s wr, 89 op/s 2024-03-09T00:09:14.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:14 smithi179 ceph-mon[36637]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 622 KiB/s wr, 90 op/s 2024-03-09T00:09:14.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:14 smithi073 ceph-mon[32309]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 622 KiB/s wr, 90 op/s 2024-03-09T00:09:14.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:14 smithi102 ceph-mon[36106]: pgmap v74: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 266 KiB/s rd, 622 KiB/s wr, 90 op/s 2024-03-09T00:09:16.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:16 smithi179 ceph-mon[36637]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-03-09T00:09:16.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:16 smithi073 ceph-mon[32309]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-03-09T00:09:16.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:16 smithi102 ceph-mon[36106]: pgmap v75: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 5.2 KiB/s wr, 4 op/s 2024-03-09T00:09:18.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:18 smithi179 ceph-mon[36637]: pgmap v76: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-09T00:09:18.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:18 smithi073 ceph-mon[32309]: pgmap v76: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-09T00:09:18.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:18 smithi102 ceph-mon[36106]: pgmap v76: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 8.3 KiB/s rd, 7.7 KiB/s wr, 5 op/s 2024-03-09T00:09:20.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:20 smithi179 ceph-mon[36637]: pgmap v77: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:20 smithi073 ceph-mon[32309]: pgmap v77: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:20.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:20 smithi102 ceph-mon[36106]: pgmap v77: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:22.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:09:22.116 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:09:22.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 -- 172.21.15.73:0/2799180666 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b381002e0 msgr2=0x7f0b3810c7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/2799180666 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b381002e0 0x7f0b3810c7f0 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 crypto rx=0x7f0b30009ca0 tx=0x7f0b3002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 -- 172.21.15.73:0/2799180666 shutdown_connections 2024-03-09T00:09:22.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/2799180666 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b381002e0 0x7f0b3810c7f0 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/2799180666 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b380ff920 0x7f0b380ffda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/2799180666 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b38105930 0x7f0b38105d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 -- 172.21.15.73:0/2799180666 >> 172.21.15.73:0/2799180666 conn(0x7f0b380767b0 msgr2=0x7f0b380fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 -- 172.21.15.73:0/2799180666 shutdown_connections 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.241+0000 7f0b3fed1640 1 -- 172.21.15.73:0/2799180666 wait complete. 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 Processor -- start 2024-03-09T00:09:22.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 -- start start 2024-03-09T00:09:22.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b380ff920 0x7f0b3819d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b381002e0 0x7f0b3819dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38111700 con 0x7f0b38105930 2024-03-09T00:09:22.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38111580 con 0x7f0b380ff920 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3fed1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b38111880 con 0x7f0b381002e0 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41730/0 (socket says 172.21.15.73:41730) 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 -- 172.21.15.73:0/3133145303 learned_addr learned my addr 172.21.15.73:0/3133145303 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b380ff920 0x7f0b3819d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b381002e0 0x7f0b3819dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 -- 172.21.15.73:0/3133145303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b381002e0 msgr2=0x7f0b3819dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b381002e0 0x7f0b3819dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 -- 172.21.15.73:0/3133145303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b380ff920 msgr2=0x7f0b3819d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.242+0000 7f0b3e46e640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b380ff920 0x7f0b3819d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b3e46e640 1 -- 172.21.15.73:0/3133145303 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b381980e0 con 0x7f0b38105930 2024-03-09T00:09:22.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b3e46e640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0b3002c700 tx=0x7f0b30032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b2effd640 1 -- 172.21.15.73:0/3133145303 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b30039040 con 0x7f0b38105930 2024-03-09T00:09:22.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b38198370 con 0x7f0b38105930 2024-03-09T00:09:22.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b2effd640 1 -- 172.21.15.73:0/3133145303 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b30044070 con 0x7f0b38105930 2024-03-09T00:09:22.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b2effd640 1 -- 172.21.15.73:0/3133145303 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b300042a0 con 0x7f0b38105930 2024-03-09T00:09:22.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.243+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b381a4670 con 0x7f0b38105930 2024-03-09T00:09:22.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.245+0000 7f0b2effd640 1 -- 172.21.15.73:0/3133145303 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0b30004440 con 0x7f0b38105930 2024-03-09T00:09:22.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.245+0000 7f0b42b90740 1 Processor -- start 2024-03-09T00:09:22.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 -- start start 2024-03-09T00:09:22.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e9fc60 0x563171ea2050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171df9e00 con 0x563171e96760 2024-03-09T00:09:22.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171da2b10 con 0x563171e9fc60 2024-03-09T00:09:22.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b42b90740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171df5ea0 con 0x563171e97080 2024-03-09T00:09:22.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b2effd640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 0x7f0b240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41738/0 (socket says 172.21.15.73:41738) 2024-03-09T00:09:22.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 -- 172.21.15.73:0/3607862668 learned_addr learned my addr 172.21.15.73:0/3607862668 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3e46e640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 0x7f0b240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e9fc60 0x563171ea2050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 -- 172.21.15.73:0/3607862668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 msgr2=0x563171e9f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 -- 172.21.15.73:0/3607862668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e9fc60 msgr2=0x563171ea2050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e9fc60 0x563171ea2050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b2effd640 1 -- 172.21.15.73:0/3133145303 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0b300f1340 con 0x7f0b38105930 2024-03-09T00:09:22.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.246+0000 7f0b3d46c640 1 -- 172.21.15.73:0/3607862668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563171bd87d0 con 0x563171e96760 2024-03-09T00:09:22.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800cfc0 tx=0x7f0b2800c5a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=561c102140617c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b3e46e640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 0x7f0b240a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0b300bd3c0 tx=0x7f0b3002c0f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b23fff640 1 -- 172.21.15.73:0/3607862668 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b28018030 con 0x563171e96760 2024-03-09T00:09:22.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b23fff640 1 -- 172.21.15.73:0/3607862668 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b28005500 con 0x563171e96760 2024-03-09T00:09:22.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b23fff640 1 -- 172.21.15.73:0/3607862668 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b2800d7b0 con 0x563171e96760 2024-03-09T00:09:22.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b42b90740 1 -- 172.21.15.73:0/3607862668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 msgr2=0x563171e96b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.247+0000 7f0b42b90740 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 crypto rx=0x7f0b2800cfc0 tx=0x7f0b2800c5a0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- 172.21.15.73:0/3607862668 shutdown_connections 2024-03-09T00:09:22.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e9fc60 0x563171ea2050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- 172.21.15.73:0/3607862668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e96760 0x563171e96b40 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- 172.21.15.73:0/3607862668 >> 172.21.15.73:0/3607862668 conn(0x563171e90e90 msgr2=0x563171e91270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- 172.21.15.73:0/3607862668 shutdown_connections 2024-03-09T00:09:22.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- 172.21.15.73:0/3607862668 wait complete. 2024-03-09T00:09:22.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- start start 2024-03-09T00:09:22.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 Processor -- start 2024-03-09T00:09:22.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171e9f410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171ea2a50 con 0x563171e9fc60 2024-03-09T00:09:22.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171db60a0 con 0x563171e96760 2024-03-09T00:09:22.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.248+0000 7f0b42b90740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563171ea2bd0 con 0x563171e97080 2024-03-09T00:09:22.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b42b90740 10 client.0 _ll_register_callbacks cb 0x563171db5a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:22.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41754/0 (socket says 172.21.15.73:41754) 2024-03-09T00:09:22.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 -- 172.21.15.73:0/1336396970 learned_addr learned my addr 172.21.15.73:0/1336396970 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171e9f410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 -- 172.21.15.73:0/1336396970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 msgr2=0x563171e9c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 -- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 msgr2=0x563171e9f410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171e9f410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 -- 172.21.15.73:0/1336396970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563171b0e3f0 con 0x563171e9fc60 2024-03-09T00:09:22.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0b280056b0 tx=0x7f0b28014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=48aa65f5336f28f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b217fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:22.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b28018030 con 0x563171e9fc60 2024-03-09T00:09:22.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b28005500 con 0x563171e9fc60 2024-03-09T00:09:22.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0b28019070 con 0x563171e9fc60 2024-03-09T00:09:22.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b28004e50 con 0x563171e9fc60 2024-03-09T00:09:22.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b42b90740 10 client.16113 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:22.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.249+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563171ace0f0 con 0x563171e9fc60 2024-03-09T00:09:22.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b28005500 con 0x563171e9fc60 2024-03-09T00:09:22.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0b28003bf0 con 0x563171e9fc60 2024-03-09T00:09:22.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 10 client.16113 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:22.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 10 client.16113 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:22.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 20 client.16113 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:22.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171eaf670 0x563171eb1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 4 client.16113 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:22.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563171e07340 con 0x563171eaf670 2024-03-09T00:09:22.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.250+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171eaf670 0x563171eb1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b217fa640 10 client.16113 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171eaf670 0x563171eb1a60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3b8faaf3bc0490e server_cookie=7129b4f18da79ea1 in_seq=0 out_seq=0 2024-03-09T00:09:22.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b217fa640 1 -- 172.21.15.73:0/1336396970 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563171e07340 con 0x563171eaf670 2024-03-09T00:09:22.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b217fa640 10 client.16113 handle_command_reply: tid=0 2024-03-09T00:09:22.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 client.16113 shutdown 2024-03-09T00:09:22.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 msgr2=0x563171e9cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f0b280056b0 tx=0x7f0b28014040 comp rx=0 tx=0).stop 2024-03-09T00:09:22.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 shutdown_connections 2024-03-09T00:09:22.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171eaf670 0x563171eb1a60 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e9fc60 0x563171e9cce0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e97080 0x563171e9c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1336396970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171e9f410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b217fa640 0 client.16113 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 >> 172.21.15.73:0/1336396970 conn(0x563171e90e90 msgr2=0x563171e92be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 shutdown_connections 2024-03-09T00:09:22.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.251+0000 7f0b42b90740 1 -- 172.21.15.73:0/1336396970 wait complete. 2024-03-09T00:09:22.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.252+0000 7f0b42b90740 20 client.16113 trim_cache size 0 max 16384 2024-03-09T00:09:22.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 Processor -- start 2024-03-09T00:09:22.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 -- start start 2024-03-09T00:09:22.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171ea7220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171e9f250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381ac750 con 0x563171e97080 2024-03-09T00:09:22.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381aed10 con 0x563171e96760 2024-03-09T00:09:22.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.266+0000 7f0b42b90740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381acd50 con 0x563171e9fc60 2024-03-09T00:09:22.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41766/0 (socket says 172.21.15.73:41766) 2024-03-09T00:09:22.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/1162353297 learned_addr learned my addr 172.21.15.73:0/1162353297 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171ea7220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171e9f250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/1162353297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 msgr2=0x563171e9f250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171e9f250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/1162353297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 msgr2=0x563171ea7220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171ea7220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/1162353297 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563171ea2ba0 con 0x563171e97080 2024-03-09T00:09:22.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171ea7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:22.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0b34004650 tx=0x7f0b3400b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=28dd574893a601ac server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b237fe640 1 -- 172.21.15.73:0/1162353297 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34003ea0 con 0x563171e97080 2024-03-09T00:09:22.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b237fe640 1 -- 172.21.15.73:0/1162353297 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b34022070 con 0x563171e97080 2024-03-09T00:09:22.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.267+0000 7f0b237fe640 1 -- 172.21.15.73:0/1162353297 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34011a80 con 0x563171e97080 2024-03-09T00:09:22.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- 172.21.15.73:0/1162353297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 msgr2=0x563171e9ed10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 crypto rx=0x7f0b34004650 tx=0x7f0b3400b390 comp rx=0 tx=0).stop 2024-03-09T00:09:22.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- 172.21.15.73:0/1162353297 shutdown_connections 2024-03-09T00:09:22.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171e9f250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171e9ed10 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 --2- 172.21.15.73:0/1162353297 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171ea7220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- 172.21.15.73:0/1162353297 >> 172.21.15.73:0/1162353297 conn(0x563171e90e90 msgr2=0x563171eab8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- 172.21.15.73:0/1162353297 shutdown_connections 2024-03-09T00:09:22.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- 172.21.15.73:0/1162353297 wait complete. 2024-03-09T00:09:22.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 -- start start 2024-03-09T00:09:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.268+0000 7f0b42b90740 1 Processor -- start 2024-03-09T00:09:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171f02dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171efd1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381ad390 con 0x563171e97080 2024-03-09T00:09:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381ae6b0 con 0x563171e96760 2024-03-09T00:09:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b381ad9f0 con 0x563171e9fc60 2024-03-09T00:09:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 10 client.0 _ll_register_callbacks cb 0x7f0b3400b690 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41768/0 (socket says 172.21.15.73:41768) 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/2089358805 learned_addr learned my addr 172.21.15.73:0/2089358805 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171f02dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/2089358805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 msgr2=0x563171efd1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171efd1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 msgr2=0x563171f02dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171f02dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 -- 172.21.15.73:0/2089358805 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563171efd8c0 con 0x563171e97080 2024-03-09T00:09:22.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b3dc6d640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x563171e8c600 tx=0x7f0b34003ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ec63bcc15e93c4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b217fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:22.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b34005440 con 0x563171e97080 2024-03-09T00:09:22.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0b34004730 con 0x563171e97080 2024-03-09T00:09:22.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b34022070 con 0x563171e97080 2024-03-09T00:09:22.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b34036050 con 0x563171e97080 2024-03-09T00:09:22.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 10 client.16119 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:22.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.269+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563171f0a540 con 0x563171e97080 2024-03-09T00:09:22.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b34022070 con 0x563171e97080 2024-03-09T00:09:22.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0b3402f060 con 0x563171e97080 2024-03-09T00:09:22.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 10 client.16119 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:22.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 10 client.16119 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:22.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 20 client.16119 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:22.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171f0aba0 0x563171f0cf90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 4 client.16119 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:09:22.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563171e01690 con 0x563171f0aba0 2024-03-09T00:09:22.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171f0aba0 0x563171f0cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b3d46c640 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171f0aba0 0x563171f0cf90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7ee5d15248bb293 server_cookie=61dbfcf95c0c3799 in_seq=0 out_seq=0 2024-03-09T00:09:22.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.270+0000 7f0b217fa640 10 client.16119 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b217fa640 1 -- 172.21.15.73:0/2089358805 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x563171e01690 con 0x563171f0aba0 2024-03-09T00:09:22.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b217fa640 10 client.16119 handle_command_reply: tid=0 2024-03-09T00:09:22.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 client.16119 shutdown 2024-03-09T00:09:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 msgr2=0x563171f03300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 crypto rx=0x563171e8c600 tx=0x7f0b34003ca0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 shutdown_connections 2024-03-09T00:09:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563171f0aba0 0x563171f0cf90 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563171e9fc60 0x563171efd1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563171e97080 0x563171f03300 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 --2- 172.21.15.73:0/2089358805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563171e96760 0x563171f02dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b217fa640 0 client.16119 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 >> 172.21.15.73:0/2089358805 conn(0x563171e90e90 msgr2=0x563171e921e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 shutdown_connections 2024-03-09T00:09:22.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 1 -- 172.21.15.73:0/2089358805 wait complete. 2024-03-09T00:09:22.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b42b90740 20 client.16119 trim_cache size 0 max 16384 2024-03-09T00:09:22.303 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:22.304 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:09:22.304 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:09:22.304 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 msgr2=0x7f0b240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 0x7f0b240a4e30 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f0b300bd3c0 tx=0x7f0b3002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 msgr2=0x7f0b38197960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 crypto rx=0x7f0b3002c700 tx=0x7f0b30032f20 comp rx=0 tx=0).stop 2024-03-09T00:09:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 shutdown_connections 2024-03-09T00:09:22.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0b240a2970 0x7f0b240a4e30 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0b38105930 0x7f0b38197960 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0b381002e0 0x7f0b3819dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 --2- 172.21.15.73:0/3133145303 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0b380ff920 0x7f0b3819d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.271+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 >> 172.21.15.73:0/3133145303 conn(0x7f0b380767b0 msgr2=0x7f0b38102780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.272+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 shutdown_connections 2024-03-09T00:09:22.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.272+0000 7f0b3fed1640 1 -- 172.21.15.73:0/3133145303 wait complete. 2024-03-09T00:09:22.309 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:09:22.310 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:09:22.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:22 smithi179 ceph-mon[36637]: pgmap v78: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 2 op/s 2024-03-09T00:09:22.334 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:22 smithi073 ceph-mon[32309]: pgmap v78: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 2 op/s 2024-03-09T00:09:22.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- 172.21.15.73:0/1691776660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070101ea0 msgr2=0x7f3070105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 --2- 172.21.15.73:0/1691776660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070101ea0 0x7f3070105ee0 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 crypto rx=0x7f3064009900 tx=0x7f306402bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- 172.21.15.73:0/1691776660 shutdown_connections 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 --2- 172.21.15.73:0/1691776660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701065c0 0x7f307010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 --2- 172.21.15.73:0/1691776660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070101ea0 0x7f3070105ee0 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 --2- 172.21.15.73:0/1691776660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30701014f0 0x7f30701018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- 172.21.15.73:0/1691776660 >> 172.21.15.73:0/1691776660 conn(0x7f30700fcd90 msgr2=0x7f30700ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- 172.21.15.73:0/1691776660 shutdown_connections 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- 172.21.15.73:0/1691776660 wait complete. 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 Processor -- start 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 -- start start 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.418+0000 7f3076ad7640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 0x7f307019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30701065c0 0x7f307019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30701116a0 con 0x7f3070197a20 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070111520 con 0x7f30701014f0 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3070111820 con 0x7f30701065c0 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41786/0 (socket says 172.21.15.73:41786) 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 -- 172.21.15.73:0/3613920081 learned_addr learned my addr 172.21.15.73:0/3613920081 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3074873640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 0x7f307019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 -- 172.21.15.73:0/3613920081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30701065c0 msgr2=0x7f307019ddb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30701065c0 0x7f307019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 -- 172.21.15.73:0/3613920081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 msgr2=0x7f307019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 0x7f307019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 -- 172.21.15.73:0/3613920081 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3070198540 con 0x7f3070197a20 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3074873640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 0x7f307019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3075074640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f306000d750 tx=0x7f306000dc10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060017070 con 0x7f3070197a20 2024-03-09T00:09:22.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3060005500 con 0x7f3070197a20 2024-03-09T00:09:22.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3070198830 con 0x7f3070197a20 2024-03-09T00:09:22.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3060004d90 con 0x7f3070197a20 2024-03-09T00:09:22.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.419+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30701a4a30 con 0x7f3070197a20 2024-03-09T00:09:22.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.420+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3070068530 con 0x7f3070197a20 2024-03-09T00:09:22.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.421+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3060004f30 con 0x7f3070197a20 2024-03-09T00:09:22.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.422+0000 7f306dffb640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 0x7f30580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.422+0000 7f3074873640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 0x7f30580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.422+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f30600cec30 con 0x7f3070197a20 2024-03-09T00:09:22.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.422+0000 7f3074873640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 0x7f30580a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3070198b60 tx=0x7f3068009520 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.426+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f306009b600 con 0x7f3070197a20 2024-03-09T00:09:22.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:22 smithi102 ceph-mon[36106]: pgmap v78: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 2.8 KiB/s wr, 2 op/s 2024-03-09T00:09:22.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.636+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3070101ea0 con 0x7f3070197a20 2024-03-09T00:09:22.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.637+0000 7f306dffb640 1 -- 172.21.15.73:0/3613920081 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f306009af00 con 0x7f3070197a20 2024-03-09T00:09:22.633 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:09:22.634 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:09:22.636 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:09:22.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 msgr2=0x7f30580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 0x7f30580a4e30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f3070198b60 tx=0x7f3068009520 comp rx=0 tx=0).stop 2024-03-09T00:09:22.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 msgr2=0x7f3070197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 crypto rx=0x7f306000d750 tx=0x7f306000dc10 comp rx=0 tx=0).stop 2024-03-09T00:09:22.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 shutdown_connections 2024-03-09T00:09:22.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f30580a2970 0x7f30580a4e30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3070197a20 0x7f3070197ed0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f30701065c0 0x7f307019ddb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 --2- 172.21.15.73:0/3613920081 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f30701014f0 0x7f307019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 >> 172.21.15.73:0/3613920081 conn(0x7f30700fcd90 msgr2=0x7f30700fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 shutdown_connections 2024-03-09T00:09:22.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.639+0000 7f3076ad7640 1 -- 172.21.15.73:0/3613920081 wait complete. 2024-03-09T00:09:22.644 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:09:22.645 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:09:22.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.794+0000 7f4853f9d640 1 -- 172.21.15.73:0/2494527397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c100d90 msgr2=0x7f484c104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.794+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2494527397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c100d90 0x7f484c104dd0 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 crypto rx=0x7f483c009900 tx=0x7f483c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.794+0000 7f4853f9d640 1 -- 172.21.15.73:0/2494527397 shutdown_connections 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.794+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2494527397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f484c1054b0 0x7f484c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2494527397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c100d90 0x7f484c104dd0 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2494527397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f484c1003e0 0x7f484c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- 172.21.15.73:0/2494527397 >> 172.21.15.73:0/2494527397 conn(0x7f484c0773a0 msgr2=0x7f484c0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- 172.21.15.73:0/2494527397 shutdown_connections 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- 172.21.15.73:0/2494527397 wait complete. 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 Processor -- start 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- start start 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f484c1054b0 0x7f484c19cb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f484c196780 0x7f484c196c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c110540 con 0x7f484c1003e0 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1103c0 con 0x7f484c196780 2024-03-09T00:09:22.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4853f9d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1106c0 con 0x7f484c1054b0 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4851d39640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4851d39640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41798/0 (socket says 172.21.15.73:41798) 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.795+0000 7f4851d39640 1 -- 172.21.15.73:0/2101246911 learned_addr learned my addr 172.21.15.73:0/2101246911 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 -- 172.21.15.73:0/2101246911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f484c1054b0 msgr2=0x7f484c19cb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f484c1054b0 0x7f484c19cb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 -- 172.21.15.73:0/2101246911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f484c196780 msgr2=0x7f484c196c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f484c196780 0x7f484c196c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 -- 172.21.15.73:0/2101246911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f484c197310 con 0x7f484c1003e0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4851d39640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f484800cce0 tx=0x7f484800a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4842ffd640 1 -- 172.21.15.73:0/2101246911 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484800f650 con 0x7f484c1003e0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4842ffd640 1 -- 172.21.15.73:0/2101246911 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f484800fcd0 con 0x7f484c1003e0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c197600 con 0x7f484c1003e0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4842ffd640 1 -- 172.21.15.73:0/2101246911 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4848018c70 con 0x7f484c1003e0 2024-03-09T00:09:22.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.796+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c1a37e0 con 0x7f484c1003e0 2024-03-09T00:09:22.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.798+0000 7f4842ffd640 1 -- 172.21.15.73:0/2101246911 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f484800f7f0 con 0x7f484c1003e0 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.798+0000 7f4856b77740 1 Processor -- start 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 -- start start 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f66e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4842ffd640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 0x7f48380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f66e590 0x55882f679760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f633360 con 0x55882f679ca0 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f5cfec0 con 0x55882f66dc70 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4856b77740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f578d30 con 0x55882f66e590 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4851d39640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 0x7f48380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41808/0 (socket says 172.21.15.73:41808) 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 -- 172.21.15.73:0/1983443961 learned_addr learned my addr 172.21.15.73:0/1983443961 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4851538640 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f66e590 0x55882f679760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f4842ffd640 1 -- 172.21.15.73:0/2101246911 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f48480cf7a0 con 0x7f484c1003e0 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f66e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 -- 172.21.15.73:0/1983443961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f66e590 msgr2=0x55882f679760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f66e590 0x55882f679760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 -- 172.21.15.73:0/1983443961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 msgr2=0x55882f66e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.799+0000 7f485253a640 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f66e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f485253a640 1 -- 172.21.15.73:0/1983443961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55882f577bc0 con 0x55882f679ca0 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4851d39640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 0x7f48380a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4848097ed0 tx=0x7f4848097e40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f485253a640 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f484400ef90 tx=0x7f48440058c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5a169d0a9ce76930 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4837fff640 1 -- 172.21.15.73:0/1983443961 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4844004020 con 0x55882f679ca0 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4837fff640 1 -- 172.21.15.73:0/1983443961 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f484400cbb0 con 0x55882f679ca0 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4837fff640 1 -- 172.21.15.73:0/1983443961 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f48440038e0 con 0x55882f679ca0 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4856b77740 1 -- 172.21.15.73:0/1983443961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 msgr2=0x55882f67c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.800+0000 7f4856b77740 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 crypto rx=0x7f484400ef90 tx=0x7f48440058c0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- 172.21.15.73:0/1983443961 shutdown_connections 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f679ca0 0x55882f67c090 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f66e590 0x55882f679760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- 172.21.15.73:0/1983443961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f66e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- 172.21.15.73:0/1983443961 >> 172.21.15.73:0/1983443961 conn(0x55882f668ae0 msgr2=0x55882f668ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- 172.21.15.73:0/1983443961 shutdown_connections 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- 172.21.15.73:0/1983443961 wait complete. 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- start start 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 Processor -- start 2024-03-09T00:09:22.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66e590 0x55882f677240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f677780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f56e4a0 con 0x55882f66dc70 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f5cbf60 con 0x55882f66e590 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.801+0000 7f4856b77740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55882f67c870 con 0x55882f679ca0 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f4856b77740 10 client.0 _ll_register_callbacks cb 0x55882f5c9d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:22.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41818/0 (socket says 172.21.15.73:41818) 2024-03-09T00:09:22.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 -- 172.21.15.73:0/1398322602 learned_addr learned my addr 172.21.15.73:0/1398322602 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f4851538640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66e590 0x55882f677240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 -- 172.21.15.73:0/1398322602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 msgr2=0x55882f677780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f677780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 -- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66e590 msgr2=0x55882f677240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66e590 0x55882f677240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 -- 172.21.15.73:0/1398322602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55882f2e43f0 con 0x55882f66dc70 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f485253a640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4844004430 tx=0x7f4844005b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=916f89891da13d53 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f48357fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f484400c8b0 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f4844013050 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f484400cbb0 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f484402e050 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f4856b77740 10 client.16137 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.802+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55882f3ae7d0 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f484400cbb0 con 0x55882f66dc70 2024-03-09T00:09:22.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f484400f040 con 0x55882f66dc70 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 10 client.16137 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 10 client.16137 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 20 client.16137 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f689b90 0x55882f68bf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 4 client.16137 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55882f5dd400 con 0x55882f689b90 2024-03-09T00:09:22.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4851538640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f689b90 0x55882f68bf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f4851538640 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f689b90 0x55882f68bf80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=50b99068ca8ad2c1 server_cookie=a1d4a11a95e9d459 in_seq=0 out_seq=0 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.803+0000 7f48357fa640 10 client.16137 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f48357fa640 1 -- 172.21.15.73:0/1398322602 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55882f5dd400 con 0x55882f689b90 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f48357fa640 10 client.16137 handle_command_reply: tid=0 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 client.16137 shutdown 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 msgr2=0x55882f676d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 crypto rx=0x7f4844004430 tx=0x7f4844005b50 comp rx=0 tx=0).stop 2024-03-09T00:09:22.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 shutdown_connections 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f689b90 0x55882f68bf80 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f677780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f48357fa640 0 client.16137 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66e590 0x55882f677240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 --2- 172.21.15.73:0/1398322602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66dc70 0x55882f676d00 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 >> 172.21.15.73:0/1398322602 conn(0x55882f668ae0 msgr2=0x55882f66a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 shutdown_connections 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 1 -- 172.21.15.73:0/1398322602 wait complete. 2024-03-09T00:09:22.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.804+0000 7f4856b77740 20 client.16137 trim_cache size 0 max 16384 2024-03-09T00:09:22.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 Processor -- start 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 -- start start 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f674730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f680af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1aa510 con 0x55882f66e590 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1acad0 con 0x55882f66dc70 2024-03-09T00:09:22.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4856b77740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1aab10 con 0x55882f679ca0 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41824/0 (socket says 172.21.15.73:41824) 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 -- 172.21.15.73:0/3518292699 learned_addr learned my addr 172.21.15.73:0/3518292699 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f485253a640 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f674730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 -- 172.21.15.73:0/3518292699 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 msgr2=0x55882f680af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f680af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 -- 172.21.15.73:0/3518292699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 msgr2=0x55882f674730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f674730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.816+0000 7f4851538640 1 -- 172.21.15.73:0/3518292699 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55882f666640 con 0x55882f66e590 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4851538640 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f483c009900 tx=0x7f483c009810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6b52c768622465b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f48377fe640 1 -- 172.21.15.73:0/3518292699 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0042c0 con 0x55882f66e590 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f48377fe640 1 -- 172.21.15.73:0/3518292699 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f483c02cd60 con 0x55882f66e590 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f48377fe640 1 -- 172.21.15.73:0/3518292699 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c004900 con 0x55882f66e590 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 -- 172.21.15.73:0/3518292699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 msgr2=0x55882f6905a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 crypto rx=0x7f483c009900 tx=0x7f483c009810 comp rx=0 tx=0).stop 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 -- 172.21.15.73:0/3518292699 shutdown_connections 2024-03-09T00:09:22.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f680af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6905a0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 --2- 172.21.15.73:0/3518292699 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f674730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 -- 172.21.15.73:0/3518292699 >> 172.21.15.73:0/3518292699 conn(0x55882f668ae0 msgr2=0x55882f666b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 -- 172.21.15.73:0/3518292699 shutdown_connections 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.817+0000 7f4856b77740 1 -- 172.21.15.73:0/3518292699 wait complete. 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 -- start start 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 Processor -- start 2024-03-09T00:09:22.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f6d93b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f6d3cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1ab150 con 0x55882f66e590 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1ac470 con 0x55882f66dc70 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c1ab7b0 con 0x55882f679ca0 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4856b77740 10 client.0 _ll_register_callbacks cb 0x7f484400e740 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41838/0 (socket says 172.21.15.73:41838) 2024-03-09T00:09:22.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 -- 172.21.15.73:0/2346489926 learned_addr learned my addr 172.21.15.73:0/2346489926 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 -- 172.21.15.73:0/2346489926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 msgr2=0x55882f6d3cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f485253a640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f6d3cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f6d3cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f485253a640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f6d93b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 -- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 msgr2=0x55882f6d93b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f6d93b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.818+0000 7f4851538640 1 -- 172.21.15.73:0/2346489926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55882f6d43f0 con 0x55882f66e590 2024-03-09T00:09:22.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4851538640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f483c006010 tx=0x7f483c02c9e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e3f0d87122641f9c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f483c0042d0 con 0x55882f66e590 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f483c04c040 con 0x55882f66e590 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f483c02cd60 con 0x55882f66e590 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f483c004470 con 0x55882f66e590 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 10 client.16143 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55882f6e0c00 con 0x55882f66e590 2024-03-09T00:09:22.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f483c02cd60 con 0x55882f66e590 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f483c045070 con 0x55882f66e590 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 10 client.16143 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 10 client.16143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 20 client.16143 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f6e1320 0x55882f6e3740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 4 client.16143 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.819+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55882f5d7750 con 0x55882f6e1320 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f485253a640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f6e1320 0x55882f6e3740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:22.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f48357fa640 10 client.16143 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f485253a640 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f6e1320 0x55882f6e3740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2949d71cb86c7163 server_cookie=e0924231c22a2f4c in_seq=0 out_seq=0 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f48357fa640 1 -- 172.21.15.73:0/2346489926 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55882f5d7750 con 0x55882f6e1320 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f48357fa640 10 client.16143 handle_command_reply: tid=0 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 client.16143 shutdown 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 msgr2=0x55882f6d98f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 crypto rx=0x7f483c006010 tx=0x7f483c02c9e0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 shutdown_connections 2024-03-09T00:09:22.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55882f6e1320 0x55882f6e3740 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55882f679ca0 0x55882f6d3cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55882f66e590 0x55882f6d98f0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f48357fa640 0 client.16143 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 --2- 172.21.15.73:0/2346489926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55882f66dc70 0x55882f6d93b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 >> 172.21.15.73:0/2346489926 conn(0x55882f668ae0 msgr2=0x55882f6905a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 shutdown_connections 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 1 -- 172.21.15.73:0/2346489926 wait complete. 2024-03-09T00:09:22.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.820+0000 7f4856b77740 20 client.16143 trim_cache size 0 max 16384 2024-03-09T00:09:22.830 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 msgr2=0x7f48380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 0x7f48380a4e30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f4848097ed0 tx=0x7f4848097e40 comp rx=0 tx=0).stop 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 msgr2=0x7f484c19c610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7f484800cce0 tx=0x7f484800a620 comp rx=0 tx=0).stop 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 shutdown_connections 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f484c196780 0x7f484c196c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f484c1054b0 0x7f484c19cb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f48380a2970 0x7f48380a4e30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 --2- 172.21.15.73:0/2101246911 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f484c1003e0 0x7f484c19c610 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:22.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 >> 172.21.15.73:0/2101246911 conn(0x7f484c0773a0 msgr2=0x7f484c0fe180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:22.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 shutdown_connections 2024-03-09T00:09:22.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:22.821+0000 7f4853f9d640 1 -- 172.21.15.73:0/2101246911 wait complete. 2024-03-09T00:09:23.400 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:23 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3613920081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:23.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:23 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3613920081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:23.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:23 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3613920081' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:23.834 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:09:23.835 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:09:23.835 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:09:23.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 -- 172.21.15.73:0/2292704984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad8100dc0 msgr2=0x7f8ad8104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2292704984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad8100dc0 0x7f8ad8104e00 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4009900 tx=0x7f8ac402bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 -- 172.21.15.73:0/2292704984 shutdown_connections 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2292704984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad81054e0 0x7f8ad810d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2292704984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad8100dc0 0x7f8ad8104e00 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2292704984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad8100410 0x7f8ad81007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.967+0000 7f8adeb97640 1 -- 172.21.15.73:0/2292704984 >> 172.21.15.73:0/2292704984 conn(0x7f8ad8076570 msgr2=0x7f8ad8076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:23.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- 172.21.15.73:0/2292704984 shutdown_connections 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- 172.21.15.73:0/2292704984 wait complete. 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 Processor -- start 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- start start 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad8100410 0x7f8ad8198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad81994e0 0x7f8ad81a3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad8110550 con 0x7f8ad81054e0 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81103d0 con 0x7f8ad81994e0 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8adeb97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81106d0 con 0x7f8ad8100410 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.968+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41856/0 (socket says 172.21.15.73:41856) 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2575089013 learned_addr learned my addr 172.21.15.73:0/2575089013 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8adc933640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad8100410 0x7f8ad8198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8add134640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad81994e0 0x7f8ad81a3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2575089013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad8100410 msgr2=0x7f8ad8198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad8100410 0x7f8ad8198a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2575089013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad81994e0 msgr2=0x7f8ad81a3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad81994e0 0x7f8ad81a3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2575089013 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ad81a4260 con 0x7f8ad81054e0 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4036040 tx=0x7f8ac402ce20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad5ffb640 1 -- 172.21.15.73:0/2575089013 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4041070 con 0x7f8ad81054e0 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ad81a44f0 con 0x7f8ad81054e0 2024-03-09T00:09:23.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad5ffb640 1 -- 172.21.15.73:0/2575089013 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ac40054e0 con 0x7f8ad81054e0 2024-03-09T00:09:23.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8ad5ffb640 1 -- 172.21.15.73:0/2575089013 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac4034440 con 0x7f8ad81054e0 2024-03-09T00:09:23.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.969+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ad81a4a30 con 0x7f8ad81054e0 2024-03-09T00:09:23.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.972+0000 7f8ad5ffb640 1 -- 172.21.15.73:0/2575089013 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8ac4044070 con 0x7f8ad81054e0 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.972+0000 7f8ae1778740 1 Processor -- start 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 -- start start 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498b160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4993d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b48eeb40 con 0x55c3b4994280 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4897d80 con 0x55c3b498ad80 2024-03-09T00:09:23.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ae1778740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b48eabe0 con 0x55c3b498b6a0 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41858/0 (socket says 172.21.15.73:41858) 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2830138741 learned_addr learned my addr 172.21.15.73:0/2830138741 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8add134640 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498b160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad5ffb640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 0x7f8ac80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8adc933640 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4993d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8add134640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 0x7f8ac80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2830138741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 msgr2=0x55c3b4993d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4993d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2830138741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 msgr2=0x55c3b498b160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498b160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad5ffb640 1 -- 172.21.15.73:0/2575089013 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8ac40f0e20 con 0x7f8ad81054e0 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.973+0000 7f8ad7fff640 1 -- 172.21.15.73:0/2830138741 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b46cb7e0 con 0x55c3b4994280 2024-03-09T00:09:23.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40bcf90 tx=0x7f8ac402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53943d7904500c4e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8add134640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 0x7f8ac80a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005fd0 tx=0x7f8ad0004ab0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ac2ffd640 1 -- 172.21.15.73:0/2830138741 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac404b070 con 0x55c3b4994280 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ac2ffd640 1 -- 172.21.15.73:0/2830138741 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ac4005690 con 0x55c3b4994280 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ac2ffd640 1 -- 172.21.15.73:0/2830138741 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ac40bd290 con 0x55c3b4994280 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ae1778740 1 -- 172.21.15.73:0/2830138741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 msgr2=0x55c3b4996670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.974+0000 7f8ae1778740 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 crypto rx=0x7f8ac40bcf90 tx=0x7f8ac402bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- 172.21.15.73:0/2830138741 shutdown_connections 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4996670 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4993d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- 172.21.15.73:0/2830138741 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498b160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- 172.21.15.73:0/2830138741 >> 172.21.15.73:0/2830138741 conn(0x55c3b49854b0 msgr2=0x55c3b4985890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- 172.21.15.73:0/2830138741 shutdown_connections 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- 172.21.15.73:0/2830138741 wait complete. 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- start start 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 Processor -- start 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b4993c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4990fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b48eeb40 con 0x55c3b4994280 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b48ab1c0 con 0x55c3b498ad80 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.975+0000 7f8ae1778740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b4996e30 con 0x55c3b498b6a0 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ae1778740 10 client.0 _ll_register_callbacks cb 0x55c3b489a7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41862/0 (socket says 172.21.15.73:41862) 2024-03-09T00:09:23.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 -- 172.21.15.73:0/1731632108 learned_addr learned my addr 172.21.15.73:0/1731632108 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8adc933640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b4993c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4990fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 -- 172.21.15.73:0/1731632108 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 msgr2=0x55c3b4990fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4990fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 -- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 msgr2=0x55c3b4993c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b4993c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 -- 172.21.15.73:0/1731632108 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3b472bf70 con 0x55c3b4994280 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8add134640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0014300 tx=0x7f8ad0004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9230b7e33030b21d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0025070 con 0x55c3b4994280 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8ad0004b20 con 0x55c3b4994280 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55c3b48c4960 con 0x55c3b4994280 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ad0030050 con 0x55c3b4994280 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ae1778740 10 client.16155 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:23.983 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.976+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c3b45c3170 con 0x55c3b4994280 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55c3b48c4960 con 0x55c3b4994280 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ad0020040 con 0x55c3b4994280 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 10 client.16155 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 10 client.16155 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 20 client.16155 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49a3fe0 0x55c3b49a63f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 4 client.16155 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3b48b0080 con 0x55c3b49a3fe0 2024-03-09T00:09:23.984 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.977+0000 7f8adc933640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49a3fe0 0x55c3b49a63f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ab7fff640 10 client.16155 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8adc933640 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49a3fe0 0x55c3b49a63f0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c5e0a878a929b7cb server_cookie=97c6de09a31db41f in_seq=0 out_seq=0 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ab7fff640 1 -- 172.21.15.73:0/1731632108 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c3b48b0080 con 0x55c3b49a3fe0 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ab7fff640 10 client.16155 handle_command_reply: tid=0 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 client.16155 shutdown 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 msgr2=0x55c3b4991520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0014300 tx=0x7f8ad0004a40 comp rx=0 tx=0).stop 2024-03-09T00:09:23.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 shutdown_connections 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49a3fe0 0x55c3b49a63f0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4991520 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b4990fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ab7fff640 0 client.16155 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 --2- 172.21.15.73:0/1731632108 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b4993c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.978+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 >> 172.21.15.73:0/1731632108 conn(0x55c3b49854b0 msgr2=0x55c3b4987200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.979+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 shutdown_connections 2024-03-09T00:09:23.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.979+0000 7f8ae1778740 1 -- 172.21.15.73:0/1731632108 wait complete. 2024-03-09T00:09:23.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.979+0000 7f8ae1778740 20 client.16155 trim_cache size 0 max 16384 2024-03-09T00:09:23.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 Processor -- start 2024-03-09T00:09:23.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 -- start start 2024-03-09T00:09:23.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b49936d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81ac7a0 con 0x55c3b4994280 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81aed60 con 0x55c3b498ad80 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8ae1778740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81acda0 con 0x55c3b498b6a0 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41868/0 (socket says 172.21.15.73:41868) 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 -- 172.21.15.73:0/762297957 learned_addr learned my addr 172.21.15.73:0/762297957 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:23.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 -- 172.21.15.73:0/762297957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 msgr2=0x55c3b49936d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b49936d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 -- 172.21.15.73:0/762297957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 msgr2=0x55c3b498ff80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.994+0000 7f8add134640 1 -- 172.21.15.73:0/762297957 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b498b4d0 con 0x55c3b4994280 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8add134640 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00142d0 tx=0x7f8ad0012eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=18137175e8ae8875 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ac27fc640 1 -- 172.21.15.73:0/762297957 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0025030 con 0x55c3b4994280 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ac27fc640 1 -- 172.21.15.73:0/762297957 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ad0009830 con 0x55c3b4994280 2024-03-09T00:09:23.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ac27fc640 1 -- 172.21.15.73:0/762297957 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad001a040 con 0x55c3b4994280 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 -- 172.21.15.73:0/762297957 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 msgr2=0x55c3b4993c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00142d0 tx=0x7f8ad0012eb0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 -- 172.21.15.73:0/762297957 shutdown_connections 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4993c10 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b49936d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 --2- 172.21.15.73:0/762297957 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b498ff80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 -- 172.21.15.73:0/762297957 >> 172.21.15.73:0/762297957 conn(0x55c3b49854b0 msgr2=0x55c3b498f870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 -- 172.21.15.73:0/762297957 shutdown_connections 2024-03-09T00:09:23.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.995+0000 7f8ae1778740 1 -- 172.21.15.73:0/762297957 wait complete. 2024-03-09T00:09:23.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 -- start start 2024-03-09T00:09:23.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 Processor -- start 2024-03-09T00:09:23.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b499adb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b499b2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81ad3e0 con 0x55c3b4994280 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81ae700 con 0x55c3b498ad80 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ad81ada40 con 0x55c3b498b6a0 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8ae1778740 10 client.0 _ll_register_callbacks cb 0x7f8ac40f4840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8add134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41878/0 (socket says 172.21.15.73:41878) 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.996+0000 7f8add134640 1 -- 172.21.15.73:0/157884613 learned_addr learned my addr 172.21.15.73:0/157884613 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b499b2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 -- 172.21.15.73:0/157884613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 msgr2=0x55c3b499b2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b499b2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 -- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 msgr2=0x55c3b499adb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b499adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 -- 172.21.15.73:0/157884613 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3b4998a20 con 0x55c3b4994280 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ad7fff640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b499b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:23.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8add134640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00127c0 tx=0x7f8ad00059b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=81dc1871aa23760e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8ad0042060 con 0x55c3b4994280 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f8ad0011a40 con 0x55c3b4994280 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8ad0009830 con 0x55c3b4994280 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8ad0049050 con 0x55c3b4994280 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ae1778740 10 client.16161 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c3b49fbfe0 con 0x55c3b4994280 2024-03-09T00:09:24.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8ad0009830 con 0x55c3b4994280 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.997+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8ad0020040 con 0x55c3b4994280 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 10 client.16161 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 10 client.16161 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 20 client.16161 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49fc570 0x55c3b49fe990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 4 client.16161 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c3b48fc080 con 0x55c3b49fc570 2024-03-09T00:09:24.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8adc933640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49fc570 0x55c3b49fe990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8adc933640 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49fc570 0x55c3b49fe990 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d3f8fb68aea28fa0 server_cookie=632a83096b592990 in_seq=0 out_seq=0 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.998+0000 7f8ab7fff640 10 client.16161 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ab7fff640 1 -- 172.21.15.73:0/157884613 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c3b48fc080 con 0x55c3b49fc570 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ab7fff640 10 client.16161 handle_command_reply: tid=0 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 client.16161 shutdown 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 msgr2=0x55c3b4998240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 crypto rx=0x7f8ad00127c0 tx=0x7f8ad00059b0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 shutdown_connections 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c3b49fc570 0x55c3b49fe990 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c3b4994280 0x55c3b4998240 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c3b498b6a0 0x55c3b499b2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 --2- 172.21.15.73:0/157884613 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c3b498ad80 0x55c3b499adb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 >> 172.21.15.73:0/157884613 conn(0x55c3b49854b0 msgr2=0x55c3b49868e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:23.999+0000 7f8ab7fff640 0 client.16161 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 shutdown_connections 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8ae1778740 1 -- 172.21.15.73:0/157884613 wait complete. 2024-03-09T00:09:24.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8ae1778740 20 client.16161 trim_cache size 0 max 16384 2024-03-09T00:09:24.004 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:24.004 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:09:24.004 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "a65bbd0e-b4c8-47e6-837b-bbfc7a324907", 2024-03-09T00:09:24.004 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:09:24.004 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:24.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 msgr2=0x7f8ac80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 0x7f8ac80a4e30 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8ad0005fd0 tx=0x7f8ad0004ab0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 msgr2=0x7f8ad8198f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 crypto rx=0x7f8ac4036040 tx=0x7f8ac402ce20 comp rx=0 tx=0).stop 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 shutdown_connections 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8ad81994e0 0x7f8ad81a3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8ac80a2970 0x7f8ac80a4e30 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8ad81054e0 0x7f8ad8198f50 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 --2- 172.21.15.73:0/2575089013 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8ad8100410 0x7f8ad8198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 >> 172.21.15.73:0/2575089013 conn(0x7f8ad8076570 msgr2=0x7f8ad80fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 shutdown_connections 2024-03-09T00:09:24.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.000+0000 7f8adeb97640 1 -- 172.21.15.73:0/2575089013 wait complete. 2024-03-09T00:09:24.008 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:09:24.008 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:09:24.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 -- 172.21.15.73:0/250647034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb544100d80 msgr2=0x7fb544104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 --2- 172.21.15.73:0/250647034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb544100d80 0x7fb544104dc0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009900 tx=0x7fb52c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:24.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 -- 172.21.15.73:0/250647034 shutdown_connections 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 --2- 172.21.15.73:0/250647034 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441054a0 0x7fb54410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 --2- 172.21.15.73:0/250647034 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb544100d80 0x7fb544104dc0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 --2- 172.21.15.73:0/250647034 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441003d0 0x7fb5441007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 -- 172.21.15.73:0/250647034 >> 172.21.15.73:0/250647034 conn(0x7fb5440772f0 msgr2=0x7fb544077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 -- 172.21.15.73:0/250647034 shutdown_connections 2024-03-09T00:09:24.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.124+0000 7fb548f16640 1 -- 172.21.15.73:0/250647034 wait complete. 2024-03-09T00:09:24.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 Processor -- start 2024-03-09T00:09:24.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 -- start start 2024-03-09T00:09:24.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441054a0 0x7fb5441023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.125 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544110510 con 0x7fb5441028e0 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544110390 con 0x7fb5441054a0 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.125+0000 7fb548f16640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb544110690 con 0x7fb5441003d0 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb54259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb54259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:34604/0 (socket says 172.21.15.73:34604) 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41896/0 (socket says 172.21.15.73:41896) 2024-03-09T00:09:24.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb54259c640 1 -- 172.21.15.73:0/2801016484 learned_addr learned my addr 172.21.15.73:0/2801016484 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb539d9b640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441054a0 0x7fb5441023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 -- 172.21.15.73:0/2801016484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 msgr2=0x7fb544101e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 -- 172.21.15.73:0/2801016484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441054a0 msgr2=0x7fb5441023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441054a0 0x7fb5441023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 -- 172.21.15.73:0/2801016484 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5441a6420 con 0x7fb5441028e0 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb54259c640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:09:24.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00ea80 tx=0x7fb53c00ef40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb53bfff640 1 -- 172.21.15.73:0/2801016484 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c00cea0 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb53bfff640 1 -- 172.21.15.73:0/2801016484 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb53c005020 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5441a6710 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb53bfff640 1 -- 172.21.15.73:0/2801016484 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c0106d0 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.126+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5441a6be0 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.129+0000 7fb53bfff640 1 -- 172.21.15.73:0/2801016484 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb53c005540 con 0x7fb5441028e0 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.129+0000 7fb54bbd0740 1 Processor -- start 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.129+0000 7fb54bbd0740 1 -- start start 2024-03-09T00:09:24.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.129+0000 7fb53bfff640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 0x7fb5300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 0x55ca55265b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 0x55ca55271050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca551c8e00 con 0x55ca5526ec60 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca55171b10 con 0x55ca55265760 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca551c4ea0 con 0x55ca55266080 2024-03-09T00:09:24.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb539d9b640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 0x7fb5300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb53bfff640 1 -- 172.21.15.73:0/2801016484 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fb53c0cf880 con 0x7fb5441028e0 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 0x55ca55271050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54259c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 0x55ca55265b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54259c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 0x55ca55265b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:46892/0 (socket says 172.21.15.73:46892) 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb54259c640 1 -- 172.21.15.73:0/2437725598 learned_addr learned my addr 172.21.15.73:0/2437725598 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb539d9b640 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 -- 172.21.15.73:0/2437725598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 msgr2=0x55ca5526e720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526e720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 -- 172.21.15.73:0/2437725598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 msgr2=0x55ca55265b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 0x55ca55265b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 -- 172.21.15.73:0/2437725598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca54fa77d0 con 0x55ca5526ec60 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb539d9b640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 0x7fb5300a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009230 tx=0x7fb52c02ca00 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.130+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 0x55ca55271050 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0ccc40 tx=0x7fb53c09ba50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3e0a751ac8f36dbe server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.131+0000 7fb538d99640 1 -- 172.21.15.73:0/2437725598 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c09bb50 con 0x55ca5526ec60 2024-03-09T00:09:24.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.131+0000 7fb538d99640 1 -- 172.21.15.73:0/2437725598 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb53c09fca0 con 0x55ca5526ec60 2024-03-09T00:09:24.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.131+0000 7fb538d99640 1 -- 172.21.15.73:0/2437725598 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb53c04a930 con 0x55ca5526ec60 2024-03-09T00:09:24.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2437725598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 msgr2=0x55ca55271050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 0x55ca55271050 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 crypto rx=0x7fb53c0ccc40 tx=0x7fb53c09ba50 comp rx=0 tx=0).stop 2024-03-09T00:09:24.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2437725598 shutdown_connections 2024-03-09T00:09:24.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca5526ec60 0x55ca55271050 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526e720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2437725598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55265760 0x55ca55265b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2437725598 >> 172.21.15.73:0/2437725598 conn(0x55ca5525fe90 msgr2=0x55ca55260270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2437725598 shutdown_connections 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2437725598 wait complete. 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- start start 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 Processor -- start 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526b6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca5526bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca552719f0 con 0x55ca55265760 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca551850a0 con 0x55ca5526ec60 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ca55271b70 con 0x55ca55266080 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.132+0000 7fb54bbd0740 10 client.0 _ll_register_callbacks cb 0x55ca55184a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41914/0 (socket says 172.21.15.73:41914) 2024-03-09T00:09:24.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 -- 172.21.15.73:0/4004684155 learned_addr learned my addr 172.21.15.73:0/4004684155 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb542d9d640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca5526bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb539d9b640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526b6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 -- 172.21.15.73:0/4004684155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 msgr2=0x55ca5526b6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526b6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 -- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 msgr2=0x55ca5526bbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca5526bbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 -- 172.21.15.73:0/4004684155 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca54edd3f0 con 0x55ca55265760 2024-03-09T00:09:24.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54259c640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb534012c80 tx=0x7fb53400f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3759676e3fc62d84 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb529ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534004050 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fb5340041f0 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb53c09fca0 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb534025050 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54bbd0740 10 client.16173 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.133+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ca54e9d0f0 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb53c09fca0 con 0x55ca55265760 2024-03-09T00:09:24.140 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb5340104c0 con 0x55ca55265760 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 10 client.16173 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 10 client.16173 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 20 client.16173 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca5527e650 0x55ca55280a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 4 client.16173 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ca551d6340 con 0x55ca5527e650 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.134+0000 7fb539d9b640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca5527e650 0x55ca55280a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.141 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.135+0000 7fb539d9b640 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca5527e650 0x55ca55280a40 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d136e4fe110f9283 server_cookie=673b7db648882abe in_seq=0 out_seq=0 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.135+0000 7fb529ffb640 10 client.16173 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.135+0000 7fb529ffb640 1 -- 172.21.15.73:0/4004684155 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ca551d6340 con 0x55ca5527e650 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.135+0000 7fb529ffb640 10 client.16173 handle_command_reply: tid=0 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 client.16173 shutdown 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 msgr2=0x55ca5526e310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 crypto rx=0x7fb534012c80 tx=0x7fb53400f790 comp rx=0 tx=0).stop 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 shutdown_connections 2024-03-09T00:09:24.142 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca5527e650 0x55ca55280a40 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca5526bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca5526b6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/4004684155 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526e310 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 >> 172.21.15.73:0/4004684155 conn(0x55ca5525fe90 msgr2=0x55ca55261be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb529ffb640 0 client.16173 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 shutdown_connections 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 1 -- 172.21.15.73:0/4004684155 wait complete. 2024-03-09T00:09:24.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.136+0000 7fb54bbd0740 20 client.16173 trim_cache size 0 max 16384 2024-03-09T00:09:24.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 Processor -- start 2024-03-09T00:09:24.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 -- start start 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55266080 0x55ca5526e310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca5526ec60 0x55ca55270050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441ae8b0 con 0x55ca55265760 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441b0e70 con 0x55ca55266080 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441aeeb0 con 0x55ca5526ec60 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41922/0 (socket says 172.21.15.73:41922) 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 -- 172.21.15.73:0/2492918245 learned_addr learned my addr 172.21.15.73:0/2492918245 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb542d9d640 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca5526ec60 0x55ca55270050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb539d9b640 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55266080 0x55ca5526e310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 -- 172.21.15.73:0/2492918245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca5526ec60 msgr2=0x55ca55270050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca5526ec60 0x55ca55270050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 -- 172.21.15.73:0/2492918245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55266080 msgr2=0x55ca5526e310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55266080 0x55ca5526e310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.153+0000 7fb54259c640 1 -- 172.21.15.73:0/2492918245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ca55285c30 con 0x55ca55265760 2024-03-09T00:09:24.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.154+0000 7fb54259c640 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c00 tx=0x7fb534010c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa94787b4895041 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.154+0000 7fb52bfff640 1 -- 172.21.15.73:0/2492918245 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534010430 con 0x55ca55265760 2024-03-09T00:09:24.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.154+0000 7fb52bfff640 1 -- 172.21.15.73:0/2492918245 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb534005590 con 0x55ca55265760 2024-03-09T00:09:24.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.154+0000 7fb52bfff640 1 -- 172.21.15.73:0/2492918245 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534003990 con 0x55ca55265760 2024-03-09T00:09:24.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2492918245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 msgr2=0x55ca5526ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 crypto rx=0x7fb534000c00 tx=0x7fb534010c90 comp rx=0 tx=0).stop 2024-03-09T00:09:24.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2492918245 shutdown_connections 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca5526ec60 0x55ca55270050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca55266080 0x55ca5526e310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/2492918245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca5526ddd0 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2492918245 >> 172.21.15.73:0/2492918245 conn(0x55ca5525fe90 msgr2=0x55ca5526b660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2492918245 shutdown_connections 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- 172.21.15.73:0/2492918245 wait complete. 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- start start 2024-03-09T00:09:24.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 Processor -- start 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca552d6b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca552d52e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441af4f0 con 0x55ca55265760 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441b0810 con 0x55ca5526ec60 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5441afb50 con 0x55ca55266080 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54bbd0740 10 client.0 _ll_register_callbacks cb 0x7fb5340128c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41924/0 (socket says 172.21.15.73:41924) 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb54259c640 1 -- 172.21.15.73:0/971276402 learned_addr learned my addr 172.21.15.73:0/971276402 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb539d9b640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca552d6b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.155+0000 7fb542d9d640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca552d52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 -- 172.21.15.73:0/971276402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 msgr2=0x55ca552d6b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca552d6b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 -- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 msgr2=0x55ca552d52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca552d52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 -- 172.21.15.73:0/971276402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ca552d59d0 con 0x55ca55265760 2024-03-09T00:09:24.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54259c640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb534005fd0 tx=0x7fb5340095b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aaf4c0dca973103 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:24.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb529ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:24.157 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:24 smithi073 ceph-mon[32309]: pgmap v79: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:24.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb534005320 con 0x55ca55265760 2024-03-09T00:09:24.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fb5340035f0 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb534005590 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb5340037f0 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54bbd0740 10 client.16179 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.156+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ca552dde60 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb534005590 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb5340326f0 con 0x55ca55265760 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 10 client.16179 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:24.160 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 10 client.16179 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 20 client.16179 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca552de410 0x55ca552e0830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 4 client.16179 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ca551d0690 con 0x55ca552de410 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb539d9b640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca552de410 0x55ca552e0830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb529ffb640 10 client.16179 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.157+0000 7fb539d9b640 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca552de410 0x55ca552e0830 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c760da18319caff0 server_cookie=56c9c82157be5dce in_seq=0 out_seq=0 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb529ffb640 1 -- 172.21.15.73:0/971276402 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55ca551d0690 con 0x55ca552de410 2024-03-09T00:09:24.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb529ffb640 10 client.16179 handle_command_reply: tid=0 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 client.16179 shutdown 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 msgr2=0x55ca552d6650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 crypto rx=0x7fb534005fd0 tx=0x7fb5340095b0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 shutdown_connections 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ca552de410 0x55ca552e0830 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ca5526ec60 0x55ca552d52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ca55266080 0x55ca552d6b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb529ffb640 0 client.16179 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 --2- 172.21.15.73:0/971276402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ca55265760 0x55ca552d6650 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 >> 172.21.15.73:0/971276402 conn(0x55ca5525fe90 msgr2=0x55ca55269f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 shutdown_connections 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 1 -- 172.21.15.73:0/971276402 wait complete. 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.158+0000 7fb54bbd0740 20 client.16179 trim_cache size 0 max 16384 2024-03-09T00:09:24.163 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2388 inodes in the stack)", 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "a65bbd0e-b4c8-47e6-837b-bbfc7a324907": { 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "a65bbd0e-b4c8-47e6-837b-bbfc7a324907", 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:09:24.164 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:09:24.165 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:24.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 msgr2=0x7fb5300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 0x7fb5300a4e30 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fb52c009230 tx=0x7fb52c02ca00 comp rx=0 tx=0).stop 2024-03-09T00:09:24.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 msgr2=0x7fb5441a5e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:24.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 crypto rx=0x7fb53c00ea80 tx=0x7fb53c00ef40 comp rx=0 tx=0).stop 2024-03-09T00:09:24.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 shutdown_connections 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb5300a2970 0x7fb5300a4e30 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb5441028e0 0x7fb5441a5e10 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb5441054a0 0x7fb5441023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 --2- 172.21.15.73:0/2801016484 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb5441003d0 0x7fb544101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 >> 172.21.15.73:0/2801016484 conn(0x7fb5440772f0 msgr2=0x7fb544068860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 shutdown_connections 2024-03-09T00:09:24.167 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:24.159+0000 7fb548f16640 1 -- 172.21.15.73:0/2801016484 wait complete. 2024-03-09T00:09:24.168 INFO:tasks.cephfs.filesystem:scrub status for tag:a65bbd0e-b4c8-47e6-837b-bbfc7a324907 - {'path': '/', 'tag': 'a65bbd0e-b4c8-47e6-837b-bbfc7a324907', 'options': 'recursive,force'} 2024-03-09T00:09:24.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:24 smithi102 ceph-mon[36106]: pgmap v79: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:24.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:24 smithi179 ceph-mon[36637]: pgmap v79: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 2.8 KiB/s wr, 3 op/s 2024-03-09T00:09:25.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:25 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:09:25.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:25 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:25.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:25 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:09:25.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:25 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:25.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:25 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:09:25.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:25 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:25.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:25 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:09:25.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:25 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:25 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:09:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:25 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:25 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:09:25.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:25 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:26.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:26 smithi073 ceph-mon[32309]: pgmap v80: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.4 KiB/s wr, 2 op/s 2024-03-09T00:09:26.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:26 smithi102 ceph-mon[36106]: pgmap v80: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.4 KiB/s wr, 2 op/s 2024-03-09T00:09:26.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:26 smithi179 ceph-mon[36637]: pgmap v80: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 2.4 KiB/s wr, 2 op/s 2024-03-09T00:09:28.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:28 smithi073 ceph-mon[32309]: pgmap v81: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 KiB/s wr, 137 op/s 2024-03-09T00:09:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:28 smithi102 ceph-mon[36106]: pgmap v81: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 KiB/s wr, 137 op/s 2024-03-09T00:09:28.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:28 smithi179 ceph-mon[36637]: pgmap v81: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 139 KiB/s rd, 4.0 KiB/s wr, 137 op/s 2024-03-09T00:09:29.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:29 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:09:29.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:29 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:09:29.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:29 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:09:30.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:30 smithi073 ceph-mon[32309]: pgmap v82: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:30.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:30 smithi102 ceph-mon[36106]: pgmap v82: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:30.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:30 smithi179 ceph-mon[36637]: pgmap v82: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:32.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:32 smithi073 ceph-mon[32309]: pgmap v83: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:32.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:32 smithi102 ceph-mon[36106]: pgmap v83: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:32.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:32 smithi179 ceph-mon[36637]: pgmap v83: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:34.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:34 smithi073 ceph-mon[32309]: pgmap v84: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:34.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:34 smithi102 ceph-mon[36106]: pgmap v84: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:34.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:34 smithi179 ceph-mon[36637]: pgmap v84: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:36.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:36 smithi073 ceph-mon[32309]: pgmap v85: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:36.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:36 smithi102 ceph-mon[36106]: pgmap v85: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:36.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:36 smithi179 ceph-mon[36637]: pgmap v85: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 288 op/s 2024-03-09T00:09:38.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:38 smithi073 ceph-mon[32309]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:38.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:38 smithi102 ceph-mon[36106]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:38.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:38 smithi179 ceph-mon[36637]: pgmap v86: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 1.6 KiB/s wr, 289 op/s 2024-03-09T00:09:40.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:40 smithi073 ceph-mon[32309]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 154 op/s 2024-03-09T00:09:40.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:40 smithi102 ceph-mon[36106]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 154 op/s 2024-03-09T00:09:40.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:40 smithi179 ceph-mon[36637]: pgmap v87: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 153 KiB/s rd, 154 op/s 2024-03-09T00:09:42.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:42 smithi073 ceph-mon[32309]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:42.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:42 smithi102 ceph-mon[36106]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:42.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:42 smithi179 ceph-mon[36637]: pgmap v88: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:43.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:43 smithi102 ceph-mon[36106]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:43.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:43 smithi179 ceph-mon[36637]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:43.651 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:43 smithi073 ceph-mon[32309]: pgmap v89: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:46.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:46 smithi179 ceph-mon[36637]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:46.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:46 smithi073 ceph-mon[32309]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:46.508 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:46 smithi102 ceph-mon[36106]: pgmap v90: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:48.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:48 smithi179 ceph-mon[36637]: pgmap v91: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:48.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:48 smithi073 ceph-mon[32309]: pgmap v91: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:48 smithi102 ceph-mon[36106]: pgmap v91: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:50.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:50 smithi179 ceph-mon[36637]: pgmap v92: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:50.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:50 smithi073 ceph-mon[32309]: pgmap v92: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:50.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:50 smithi102 ceph-mon[36106]: pgmap v92: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:52.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:52 smithi179 ceph-mon[36637]: pgmap v93: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:52.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:52 smithi073 ceph-mon[32309]: pgmap v93: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:52 smithi102 ceph-mon[36106]: pgmap v93: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:54.169 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:09:54.169 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:09:54.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- 172.21.15.73:0/2484139009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c068740 msgr2=0x7f401c068b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 --2- 172.21.15.73:0/2484139009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c068740 0x7f401c068b20 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 crypto rx=0x7f4010009900 tx=0x7f401002bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:54.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- 172.21.15.73:0/2484139009 shutdown_connections 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 --2- 172.21.15.73:0/2484139009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 0x7f401c10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 --2- 172.21.15.73:0/2484139009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c0690f0 0x7f401c105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 --2- 172.21.15.73:0/2484139009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c068740 0x7f401c068b20 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- 172.21.15.73:0/2484139009 >> 172.21.15.73:0/2484139009 conn(0x7f401c0772f0 msgr2=0x7f401c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- 172.21.15.73:0/2484139009 shutdown_connections 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- 172.21.15.73:0/2484139009 wait complete. 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 Processor -- start 2024-03-09T00:09:54.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.293+0000 7f4022a73640 1 -- start start 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c068740 0x7f401c19d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 0x7f401c19dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c111660 con 0x7f401c1979c0 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1114e0 con 0x7f401c068740 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1117e0 con 0x7f401c1065a0 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56838/0 (socket says 172.21.15.73:56838) 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 -- 172.21.15.73:0/2015661908 learned_addr learned my addr 172.21.15.73:0/2015661908 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f402080f640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c068740 0x7f401c19d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f401bfff640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 0x7f401c19dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 -- 172.21.15.73:0/2015661908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 msgr2=0x7f401c19dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 0x7f401c19dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 -- 172.21.15.73:0/2015661908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c068740 msgr2=0x7f401c19d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c068740 0x7f401c19d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 -- 172.21.15.73:0/2015661908 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f401c1984e0 con 0x7f401c1979c0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4021010640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f401400cad0 tx=0x7f401400cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4019ffb640 1 -- 172.21.15.73:0/2015661908 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400ae20 con 0x7f401c1979c0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4019ffb640 1 -- 172.21.15.73:0/2015661908 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4014005020 con 0x7f401c1979c0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f401c1987d0 con 0x7f401c1979c0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.294+0000 7f4019ffb640 1 -- 172.21.15.73:0/2015661908 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401400f6f0 con 0x7f401c1979c0 2024-03-09T00:09:54.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.295+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f401c1a4a20 con 0x7f401c1979c0 2024-03-09T00:09:54.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4019ffb640 1 -- 172.21.15.73:0/2015661908 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f40140051d0 con 0x7f401c1979c0 2024-03-09T00:09:54.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4025652740 1 Processor -- start 2024-03-09T00:09:54.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4019ffb640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 0x7f400c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4025652740 1 -- start start 2024-03-09T00:09:54.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4025652740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f401bfff640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 0x7f400c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.297+0000 7f4025652740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 0x556d3291f720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4025652740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d32922050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4019ffb640 1 -- 172.21.15.73:0/2015661908 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f40140cf470 con 0x7f401c1979c0 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4025652740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d32879e00 con 0x556d32916760 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4025652740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d32822b10 con 0x556d32917080 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4025652740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d32875ea0 con 0x556d3291fc60 2024-03-09T00:09:54.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56840/0 (socket says 172.21.15.73:56840) 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 -- 172.21.15.73:0/558045138 learned_addr learned my addr 172.21.15.73:0/558045138 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4021010640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d32922050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f401bfff640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 0x556d3291f720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 -- 172.21.15.73:0/558045138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 msgr2=0x556d32922050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d32922050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 -- 172.21.15.73:0/558045138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 msgr2=0x556d3291f720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 0x556d3291f720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 -- 172.21.15.73:0/558045138 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d326587d0 con 0x556d32916760 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f401bfff640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 0x556d3291f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f401bfff640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 0x7f400c0a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ffca0 tx=0x7f400800b650 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f402080f640 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f401002c170 tx=0x7f4010005c00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5b19e9a0980c4f96 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4006ffd640 1 -- 172.21.15.73:0/558045138 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010042070 con 0x556d32916760 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4006ffd640 1 -- 172.21.15.73:0/558045138 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f401002ca80 con 0x556d32916760 2024-03-09T00:09:54.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.298+0000 7f4006ffd640 1 -- 172.21.15.73:0/558045138 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4010005180 con 0x556d32916760 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- 172.21.15.73:0/558045138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 msgr2=0x556d32916b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 crypto rx=0x7f401002c170 tx=0x7f4010005c00 comp rx=0 tx=0).stop 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- 172.21.15.73:0/558045138 shutdown_connections 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d32922050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32917080 0x556d3291f720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 --2- 172.21.15.73:0/558045138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d32916b40 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- 172.21.15.73:0/558045138 >> 172.21.15.73:0/558045138 conn(0x556d32910e90 msgr2=0x556d32911270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- 172.21.15.73:0/558045138 shutdown_connections 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- 172.21.15.73:0/558045138 wait complete. 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 -- start start 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.299+0000 7f4025652740 1 Processor -- start 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d3291c760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 0x556d3291cca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d329229f0 con 0x556d32916760 2024-03-09T00:09:54.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d328360a0 con 0x556d3291fc60 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556d32922b70 con 0x556d32917080 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4025652740 10 client.0 _ll_register_callbacks cb 0x556d32835a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56848/0 (socket says 172.21.15.73:56848) 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 -- 172.21.15.73:0/3215671828 learned_addr learned my addr 172.21.15.73:0/3215671828 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f401bfff640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d3291c760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4021010640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 0x556d3291cca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 -- 172.21.15.73:0/3215671828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 msgr2=0x556d3291c760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d3291c760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 -- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 msgr2=0x556d3291cca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 0x556d3291cca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 -- 172.21.15.73:0/3215671828 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d3258e3f0 con 0x556d32916760 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f401bfff640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d3291c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f4021010640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 0x556d3291cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f402080f640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f401002bc70 tx=0x7f40100056f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b93f72810414447 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.300+0000 7f3ffbfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:54.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401004a070 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f40100046c0 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f401002ca80 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4010056050 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 10 client.16191 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556d3254e0f0 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f401002ca80 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f401003e520 con 0x556d32916760 2024-03-09T00:09:54.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 10 client.16191 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 10 client.16191 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 20 client.16191 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d3292f4c0 0x556d329318b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 4 client.16191 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.301+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556d32887340 con 0x556d3292f4c0 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.302+0000 7f401bfff640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d3292f4c0 0x556d329318b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.302+0000 7f401bfff640 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d3292f4c0 0x556d329318b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3388f9c9fd94d5a server_cookie=bb916a7fa18b1697 in_seq=0 out_seq=0 2024-03-09T00:09:54.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.302+0000 7f3ffbfff640 10 client.16191 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f3ffbfff640 1 -- 172.21.15.73:0/3215671828 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556d32887340 con 0x556d3292f4c0 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f3ffbfff640 10 client.16191 handle_command_reply: tid=0 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 client.16191 shutdown 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 msgr2=0x556d3291f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 crypto rx=0x7f401002bc70 tx=0x7f40100056f0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 shutdown_connections 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d3292f4c0 0x556d329318b0 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d3291fc60 0x556d3291cca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f3ffbfff640 0 client.16191 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d3291c760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 --2- 172.21.15.73:0/3215671828 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32916760 0x556d3291f3d0 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 >> 172.21.15.73:0/3215671828 conn(0x556d32910e90 msgr2=0x556d32912be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 shutdown_connections 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 1 -- 172.21.15.73:0/3215671828 wait complete. 2024-03-09T00:09:54.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.303+0000 7f4025652740 20 client.16191 trim_cache size 0 max 16384 2024-03-09T00:09:54.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 Processor -- start 2024-03-09T00:09:54.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 -- start start 2024-03-09T00:09:54.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d3291ee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d3291b030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ac6c0 con 0x556d32917080 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1aec80 con 0x556d32916760 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4025652740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1accc0 con 0x556d3291fc60 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56860/0 (socket says 172.21.15.73:56860) 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 -- 172.21.15.73:0/816460865 learned_addr learned my addr 172.21.15.73:0/816460865 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f4021010640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d3291b030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 -- 172.21.15.73:0/816460865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 msgr2=0x556d3291b030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d3291b030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f402080f640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d3291ee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 -- 172.21.15.73:0/816460865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 msgr2=0x556d3291ee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d3291ee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f401bfff640 1 -- 172.21.15.73:0/816460865 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556d32936760 con 0x556d32917080 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.318+0000 7f402080f640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d3291ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f401bfff640 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f401c1991b0 tx=0x7f4008014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3dfeff6fe86ce944 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f40067fc640 1 -- 172.21.15.73:0/816460865 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008028290 con 0x556d32917080 2024-03-09T00:09:54.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f40067fc640 1 -- 172.21.15.73:0/816460865 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4008020430 con 0x556d32917080 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f40067fc640 1 -- 172.21.15.73:0/816460865 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4008030640 con 0x556d32917080 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 -- 172.21.15.73:0/816460865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 msgr2=0x556d3291f3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 crypto rx=0x7f401c1991b0 tx=0x7f4008014000 comp rx=0 tx=0).stop 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 -- 172.21.15.73:0/816460865 shutdown_connections 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d3291fc60 0x556d3291b030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d32917080 0x556d3291f3d0 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 --2- 172.21.15.73:0/816460865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d3291ee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 -- 172.21.15.73:0/816460865 >> 172.21.15.73:0/816460865 conn(0x556d32910e90 msgr2=0x556d3291c720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 -- 172.21.15.73:0/816460865 shutdown_connections 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.319+0000 7f4025652740 1 -- 172.21.15.73:0/816460865 wait complete. 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 -- start start 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 Processor -- start 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d32926e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d32923380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ad300 con 0x556d3291fc60 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ae620 con 0x556d32916760 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f401c1ad960 con 0x556d32917080 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4025652740 10 client.0 _ll_register_callbacks cb 0x7f401002c5d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4021010640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4021010640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56870/0 (socket says 172.21.15.73:56870) 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f4021010640 1 -- 172.21.15.73:0/911238719 learned_addr learned my addr 172.21.15.73:0/911238719 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.320+0000 7f401bfff640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d32923380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f402080f640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d32926e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 -- 172.21.15.73:0/911238719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 msgr2=0x556d32923380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d32923380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 -- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 msgr2=0x556d32926e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d32926e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 -- 172.21.15.73:0/911238719 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556d32923e00 con 0x556d3291fc60 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4021010640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40140cc840 tx=0x7f401409bee0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=246f850d99530713 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f401401d040 con 0x556d3291fc60 2024-03-09T00:09:54.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f40140860b0 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4008020430 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40140a3f80 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 10 client.16197 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556d32986320 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4008020430 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f40140cf140 con 0x556d3291fc60 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 10 client.16197 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:54.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 10 client.16197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 20 client.16197 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d32986b20 0x556d32988f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 4 client.16197 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.321+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x556d32881690 con 0x556d32986b20 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f402080f640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d32986b20 0x556d32988f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f402080f640 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d32986b20 0x556d32988f10 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=640d8144d50f1a2c server_cookie=e3ea30f0b8c0f2f0 in_seq=0 out_seq=0 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f3ffbfff640 10 client.16197 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f3ffbfff640 1 -- 172.21.15.73:0/911238719 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x556d32881690 con 0x556d32986b20 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f3ffbfff640 10 client.16197 handle_command_reply: tid=0 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 client.16197 shutdown 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 msgr2=0x556d329238c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 crypto rx=0x7f40140cc840 tx=0x7f401409bee0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 shutdown_connections 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556d32986b20 0x556d32988f10 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556d3291fc60 0x556d329238c0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f3ffbfff640 0 client.16197 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556d32917080 0x556d32923380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 --2- 172.21.15.73:0/911238719 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556d32916760 0x556d32926e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 >> 172.21.15.73:0/911238719 conn(0x556d32910e90 msgr2=0x556d3291af50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 shutdown_connections 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.322+0000 7f4025652740 1 -- 172.21.15.73:0/911238719 wait complete. 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4025652740 20 client.16197 trim_cache size 0 max 16384 2024-03-09T00:09:54.329 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:54.330 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:09:54.330 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:09:54.330 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 msgr2=0x7f400c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 0x7f400c0a4e30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f401c0ffca0 tx=0x7f400800b650 comp rx=0 tx=0).stop 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 msgr2=0x7f401c197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 crypto rx=0x7f401400cad0 tx=0x7f401400cf90 comp rx=0 tx=0).stop 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 shutdown_connections 2024-03-09T00:09:54.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f400c0a2970 0x7f400c0a4e30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f401c1979c0 0x7f401c197e70 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f401c1065a0 0x7f401c19dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 --2- 172.21.15.73:0/2015661908 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f401c068740 0x7f401c19d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 >> 172.21.15.73:0/2015661908 conn(0x7f401c0772f0 msgr2=0x7f401c0fe330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 shutdown_connections 2024-03-09T00:09:54.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.323+0000 7f4022a73640 1 -- 172.21.15.73:0/2015661908 wait complete. 2024-03-09T00:09:54.333 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:54 smithi073 ceph-mon[32309]: pgmap v94: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:54.334 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:09:54.334 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:09:54.338 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:54 smithi179 ceph-mon[36637]: pgmap v94: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:54.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 -- 172.21.15.73:0/3117077701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc41014f0 msgr2=0x7f3bc41018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 --2- 172.21.15.73:0/3117077701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc41014f0 0x7f3bc41018d0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 crypto rx=0x7f3bb8009900 tx=0x7f3bb802bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:54.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 -- 172.21.15.73:0/3117077701 shutdown_connections 2024-03-09T00:09:54.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 --2- 172.21.15.73:0/3117077701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 0x7f3bc410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 --2- 172.21.15.73:0/3117077701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc4101ea0 0x7f3bc4105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 --2- 172.21.15.73:0/3117077701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc41014f0 0x7f3bc41018d0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 -- 172.21.15.73:0/3117077701 >> 172.21.15.73:0/3117077701 conn(0x7f3bc40fcd90 msgr2=0x7f3bc40ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 -- 172.21.15.73:0/3117077701 shutdown_connections 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.444+0000 7f3bcad62640 1 -- 172.21.15.73:0/3117077701 wait complete. 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 Processor -- start 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 -- start start 2024-03-09T00:09:54.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc41014f0 0x7f3bc419d850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 0x7f3bc419dd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc4111670 con 0x7f3bc4197a50 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc41114f0 con 0x7f3bc41065c0 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bcad62640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bc41117f0 con 0x7f3bc41014f0 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56892/0 (socket says 172.21.15.73:56892) 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 -- 172.21.15.73:0/500735749 learned_addr learned my addr 172.21.15.73:0/500735749 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc8afe640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc41014f0 0x7f3bc419d850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc3fff640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 0x7f3bc419dd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 -- 172.21.15.73:0/500735749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc41014f0 msgr2=0x7f3bc419d850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc41014f0 0x7f3bc419d850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 -- 172.21.15.73:0/500735749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 msgr2=0x7f3bc419dd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 0x7f3bc419dd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.445+0000 7f3bc92ff640 1 -- 172.21.15.73:0/500735749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bc4198570 con 0x7f3bc4197a50 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bc92ff640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00cad0 tx=0x7f3bbc00cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00ae20 con 0x7f3bc4197a50 2024-03-09T00:09:54.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bbc005020 con 0x7f3bc4197a50 2024-03-09T00:09:54.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bc4198860 con 0x7f3bc4197a50 2024-03-09T00:09:54.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bbc00f6f0 con 0x7f3bc4197a50 2024-03-09T00:09:54.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.446+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bc41a4a10 con 0x7f3bc4197a50 2024-03-09T00:09:54.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.447+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bc4068530 con 0x7f3bc4197a50 2024-03-09T00:09:54.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.448+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3bbc0051d0 con 0x7f3bc4197a50 2024-03-09T00:09:54.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.448+0000 7f3bc1ffb640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 0x7f3bb40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.448+0000 7f3bc8afe640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 0x7f3bb40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.448+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f3bbc0cf150 con 0x7f3bc4197a50 2024-03-09T00:09:54.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.449+0000 7f3bc8afe640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 0x7f3bb40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3bc41004d0 tx=0x7f3bb8004f10 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.453+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3bbc014030 con 0x7f3bc4197a50 2024-03-09T00:09:54.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:54 smithi102 ceph-mon[36106]: pgmap v94: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:09:54.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.664+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bc4101ea0 con 0x7f3bc4197a50 2024-03-09T00:09:54.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.665+0000 7f3bc1ffb640 1 -- 172.21.15.73:0/500735749 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f3bbc09b4d0 con 0x7f3bc4197a50 2024-03-09T00:09:54.660 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:09:54.660 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:09:54.660 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 msgr2=0x7f3bb40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 0x7f3bb40a4e30 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3bc41004d0 tx=0x7f3bb8004f10 comp rx=0 tx=0).stop 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 msgr2=0x7f3bc4197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc00cad0 tx=0x7f3bbc00cf90 comp rx=0 tx=0).stop 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 shutdown_connections 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f3bb40a2970 0x7f3bb40a4e30 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3bc4197a50 0x7f3bc4197f00 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3bc41065c0 0x7f3bc419dd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 --2- 172.21.15.73:0/500735749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3bc41014f0 0x7f3bc419d850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 >> 172.21.15.73:0/500735749 conn(0x7f3bc40fcd90 msgr2=0x7f3bc40fe140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 shutdown_connections 2024-03-09T00:09:54.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.667+0000 7f3bcad62640 1 -- 172.21.15.73:0/500735749 wait complete. 2024-03-09T00:09:54.673 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:09:54.673 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:09:54.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 -- 172.21.15.73:0/3460126072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 msgr2=0x7fd81010d000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 --2- 172.21.15.73:0/3460126072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81010d000 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 crypto rx=0x7fd808009ca0 tx=0x7fd80802c140 comp rx=0 tx=0).stop 2024-03-09T00:09:54.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 -- 172.21.15.73:0/3460126072 shutdown_connections 2024-03-09T00:09:54.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 --2- 172.21.15.73:0/3460126072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81010d000 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 --2- 172.21.15.73:0/3460126072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd810100440 0x7fd8101008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 --2- 172.21.15.73:0/3460126072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd8101062d0 0x7fd8101066b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 -- 172.21.15.73:0/3460126072 >> 172.21.15.73:0/3460126072 conn(0x7fd810076fd0 msgr2=0x7fd8100773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 -- 172.21.15.73:0/3460126072 shutdown_connections 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.782+0000 7fd81868c640 1 -- 172.21.15.73:0/3460126072 wait complete. 2024-03-09T00:09:54.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 Processor -- start 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 -- start start 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd810100440 0x7fd81019f900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8101062d0 0x7fd810199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101105b0 con 0x7fd810100e00 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810110430 con 0x7fd8101062d0 2024-03-09T00:09:54.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd81868c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd810110730 con 0x7fd810100440 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56902/0 (socket says 172.21.15.73:56902) 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 -- 172.21.15.73:0/2468721777 learned_addr learned my addr 172.21.15.73:0/2468721777 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd816c29640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8101062d0 0x7fd810199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd816428640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd810100440 0x7fd81019f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 -- 172.21.15.73:0/2468721777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd810100440 msgr2=0x7fd81019f900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd810100440 0x7fd81019f900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 -- 172.21.15.73:0/2468721777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8101062d0 msgr2=0x7fd810199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8101062d0 0x7fd810199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 -- 172.21.15.73:0/2468721777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd81019a230 con 0x7fd810100e00 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.783+0000 7fd815c27640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c890 tx=0x7fd80000cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.784+0000 7fd8077fe640 1 -- 172.21.15.73:0/2468721777 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800004220 con 0x7fd810100e00 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.784+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd81019a520 con 0x7fd810100e00 2024-03-09T00:09:54.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.784+0000 7fd8077fe640 1 -- 172.21.15.73:0/2468721777 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8000043c0 con 0x7fd810100e00 2024-03-09T00:09:54.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.784+0000 7fd8077fe640 1 -- 172.21.15.73:0/2468721777 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800004a00 con 0x7fd810100e00 2024-03-09T00:09:54.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.784+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8101a67c0 con 0x7fd810100e00 2024-03-09T00:09:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd8077fe640 1 -- 172.21.15.73:0/2468721777 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd8000079b0 con 0x7fd810100e00 2024-03-09T00:09:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 Processor -- start 2024-03-09T00:09:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 -- start start 2024-03-09T00:09:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e704580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 0x55ec0e70d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e6cb360 con 0x55ec0e70d6a0 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e667ec0 con 0x55ec0e704ac0 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.786+0000 7fd81b274740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e610d30 con 0x55ec0e7041a0 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd8077fe640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 0x7fd7fc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56916/0 (socket says 172.21.15.73:56916) 2024-03-09T00:09:54.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd816c29640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 0x7fd7fc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 -- 172.21.15.73:0/915405628 learned_addr learned my addr 172.21.15.73:0/915405628 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd816428640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 0x55ec0e70d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd816c29640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e704580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd8077fe640 1 -- 172.21.15.73:0/2468721777 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd8000ceec0 con 0x7fd810100e00 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 -- 172.21.15.73:0/915405628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 msgr2=0x55ec0e704580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e704580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 -- 172.21.15.73:0/915405628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 msgr2=0x55ec0e70d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 0x55ec0e70d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 -- 172.21.15.73:0/915405628 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0e60fbc0 con 0x55ec0e70d6a0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd816428640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 0x55ec0e70d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd816c29640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 0x7fd7fc0a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd808004af0 tx=0x7fd808009a20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd815c27640 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd80005c000 tx=0x7fd80009b090 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95fa894d0eb07ca4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd804ff9640 1 -- 172.21.15.73:0/915405628 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80001d070 con 0x55ec0e70d6a0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd804ff9640 1 -- 172.21.15.73:0/915405628 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80009b190 con 0x55ec0e70d6a0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.787+0000 7fd804ff9640 1 -- 172.21.15.73:0/915405628 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800007ca0 con 0x55ec0e70d6a0 2024-03-09T00:09:54.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- 172.21.15.73:0/915405628 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 msgr2=0x55ec0e70fa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 crypto rx=0x7fd80005c000 tx=0x7fd80009b090 comp rx=0 tx=0).stop 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- 172.21.15.73:0/915405628 shutdown_connections 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70fa90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e704ac0 0x55ec0e70d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 --2- 172.21.15.73:0/915405628 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e704580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- 172.21.15.73:0/915405628 >> 172.21.15.73:0/915405628 conn(0x55ec0e6fe8d0 msgr2=0x55ec0e6fecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- 172.21.15.73:0/915405628 shutdown_connections 2024-03-09T00:09:54.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- 172.21.15.73:0/915405628 wait complete. 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 -- start start 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.788+0000 7fd81b274740 1 Processor -- start 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e704ac0 0x55ec0e70ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70b0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e6064a0 con 0x55ec0e7041a0 2024-03-09T00:09:54.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e663f60 con 0x55ec0e70d6a0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ec0e710430 con 0x55ec0e704ac0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 10 client.0 _ll_register_callbacks cb 0x55ec0e661d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56922/0 (socket says 172.21.15.73:56922) 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 -- 172.21.15.73:0/3711885564 learned_addr learned my addr 172.21.15.73:0/3711885564 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816c29640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70b0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd815c27640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e704ac0 0x55ec0e70ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 -- 172.21.15.73:0/3711885564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e704ac0 msgr2=0x55ec0e70ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e704ac0 0x55ec0e70ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 -- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 msgr2=0x55ec0e70b0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70b0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 -- 172.21.15.73:0/3711885564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0e37c3f0 con 0x55ec0e7041a0 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816c29640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd816428640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00bf10 tx=0x7fd80c012820 comp rx=0 tx=0).ready entity=mon.0 client_cookie=900ceed1c1f4de02 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd7f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:54.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80c004230 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd80c0043d0 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd80009b190 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd80c010760 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.789+0000 7fd81b274740 10 client.16215 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ec0e4467d0 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd80009b190 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd80c003c90 con 0x55ec0e7041a0 2024-03-09T00:09:54.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 10 client.16215 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 10 client.16215 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 20 client.16215 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e71d880 0x55ec0e71fcd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 4 client.16215 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ec0e675400 con 0x55ec0e71d880 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd815c27640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e71d880 0x55ec0e71fcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd7f9ffb640 10 client.16215 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.790+0000 7fd815c27640 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e71d880 0x55ec0e71fcd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7f7f18ed6480882d server_cookie=367979d568763364 in_seq=0 out_seq=0 2024-03-09T00:09:54.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/3711885564 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ec0e675400 con 0x55ec0e71d880 2024-03-09T00:09:54.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd7f9ffb640 10 client.16215 handle_command_reply: tid=0 2024-03-09T00:09:54.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 client.16215 shutdown 2024-03-09T00:09:54.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 msgr2=0x55ec0e70a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 crypto rx=0x7fd80c00bf10 tx=0x7fd80c012820 comp rx=0 tx=0).stop 2024-03-09T00:09:54.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 shutdown_connections 2024-03-09T00:09:54.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e71d880 0x55ec0e71fcd0 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e70b0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd7f9ffb640 0 client.16215 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e704ac0 0x55ec0e70ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 --2- 172.21.15.73:0/3711885564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e7041a0 0x55ec0e70a640 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 >> 172.21.15.73:0/3711885564 conn(0x55ec0e6fe8d0 msgr2=0x55ec0e700620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 shutdown_connections 2024-03-09T00:09:54.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 1 -- 172.21.15.73:0/3711885564 wait complete. 2024-03-09T00:09:54.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.791+0000 7fd81b274740 20 client.16215 trim_cache size 0 max 16384 2024-03-09T00:09:54.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 Processor -- start 2024-03-09T00:09:54.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 -- start start 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e70cd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e707e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101ae850 con 0x55ec0e704ac0 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b0e10 con 0x55ec0e70d6a0 2024-03-09T00:09:54.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd81b274740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101aee50 con 0x55ec0e7041a0 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56934/0 (socket says 172.21.15.73:56934) 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 -- 172.21.15.73:0/532737821 learned_addr learned my addr 172.21.15.73:0/532737821 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd816428640 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e70cd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 -- 172.21.15.73:0/532737821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 msgr2=0x55ec0e70cd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e70cd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 -- 172.21.15.73:0/532737821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 msgr2=0x55ec0e707e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e707e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 -- 172.21.15.73:0/532737821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ec0e724ec0 con 0x55ec0e704ac0 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.803+0000 7fd815c27640 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd80009b120 tx=0x7fd80001a040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5689c87e056dcdc9 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd7fbfff640 1 -- 172.21.15.73:0/532737821 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80001d040 con 0x55ec0e704ac0 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd7fbfff640 1 -- 172.21.15.73:0/532737821 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd800085e50 con 0x55ec0e704ac0 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd7fbfff640 1 -- 172.21.15.73:0/532737821 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd800097800 con 0x55ec0e704ac0 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- 172.21.15.73:0/532737821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 msgr2=0x55ec0e7078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 crypto rx=0x7fd80009b120 tx=0x7fd80001a040 comp rx=0 tx=0).stop 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- 172.21.15.73:0/532737821 shutdown_connections 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e70d6a0 0x55ec0e707e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e7078c0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 --2- 172.21.15.73:0/532737821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e7041a0 0x55ec0e70cd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- 172.21.15.73:0/532737821 >> 172.21.15.73:0/532737821 conn(0x55ec0e6fe8d0 msgr2=0x55ec0e704ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- 172.21.15.73:0/532737821 shutdown_connections 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- 172.21.15.73:0/532737821 wait complete. 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.804+0000 7fd81b274740 1 -- start start 2024-03-09T00:09:54.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 Processor -- start 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 0x55ec0e714ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e70d6a0 0x55ec0e713470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101af490 con 0x55ec0e704ac0 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101b07b0 con 0x55ec0e7041a0 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8101afaf0 con 0x55ec0e70d6a0 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd81b274740 10 client.0 _ll_register_callbacks cb 0x7fd80009afe0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:54.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56944/0 (socket says 172.21.15.73:56944) 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 -- 172.21.15.73:0/2799696827 learned_addr learned my addr 172.21.15.73:0/2799696827 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd816428640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 0x55ec0e714ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd816c29640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e70d6a0 0x55ec0e713470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 -- 172.21.15.73:0/2799696827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e70d6a0 msgr2=0x55ec0e713470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e70d6a0 0x55ec0e713470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 -- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 msgr2=0x55ec0e714ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:54.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 0x55ec0e714ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 -- 172.21.15.73:0/2799696827 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ec0e713ca0 con 0x55ec0e704ac0 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd816428640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 0x55ec0e714ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd815c27640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd8000975f0 tx=0x7fd8000870d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8a00daf4544c7fa1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd7f9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.805+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd80001d070 con 0x55ec0e704ac0 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd800099490 con 0x55ec0e704ac0 2024-03-09T00:09:54.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd800085e50 con 0x55ec0e704ac0 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd80009f2e0 con 0x55ec0e704ac0 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 10 client.16221 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ec0e774e90 con 0x55ec0e704ac0 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd800085e50 con 0x55ec0e704ac0 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd800007ca0 con 0x55ec0e704ac0 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 10 client.16221 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 10 client.16221 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:54.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 20 client.16221 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e775060 0x55ec0e777450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 4 client.16221 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55ec0e66f750 con 0x55ec0e775060 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.806+0000 7fd816428640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e775060 0x55ec0e777450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd816428640 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e775060 0x55ec0e777450 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=82e4cd196b196608 server_cookie=7c532b6bff4498fd in_seq=0 out_seq=0 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd7f9ffb640 10 client.16221 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd7f9ffb640 1 -- 172.21.15.73:0/2799696827 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55ec0e66f750 con 0x55ec0e775060 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd7f9ffb640 10 client.16221 handle_command_reply: tid=0 2024-03-09T00:09:54.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 client.16221 shutdown 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 msgr2=0x55ec0e712ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 crypto rx=0x7fd8000975f0 tx=0x7fd8000870d0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 shutdown_connections 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ec0e775060 0x55ec0e777450 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ec0e70d6a0 0x55ec0e713470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ec0e704ac0 0x55ec0e712ed0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd7f9ffb640 0 client.16221 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:54.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 --2- 172.21.15.73:0/2799696827 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ec0e7041a0 0x55ec0e714ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 >> 172.21.15.73:0/2799696827 conn(0x55ec0e6fe8d0 msgr2=0x55ec0e708f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 shutdown_connections 2024-03-09T00:09:54.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 1 -- 172.21.15.73:0/2799696827 wait complete. 2024-03-09T00:09:54.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81b274740 20 client.16221 trim_cache size 0 max 16384 2024-03-09T00:09:54.832 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 msgr2=0x7fd7fc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 0x7fd7fc0a4e30 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7fd808004af0 tx=0x7fd808009a20 comp rx=0 tx=0).stop 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 msgr2=0x7fd81019fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 crypto rx=0x7fd80000c890 tx=0x7fd80000cd50 comp rx=0 tx=0).stop 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 shutdown_connections 2024-03-09T00:09:54.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd8101062d0 0x7fd810199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.807+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7fc0a2970 0x7fd7fc0a4e30 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.808+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd810100e00 0x7fd81019fe40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.808+0000 7fd81868c640 1 --2- 172.21.15.73:0/2468721777 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd810100440 0x7fd81019f900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.808+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 >> 172.21.15.73:0/2468721777 conn(0x7fd810076fd0 msgr2=0x7fd810069990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.808+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 shutdown_connections 2024-03-09T00:09:54.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:54.808+0000 7fd81868c640 1 -- 172.21.15.73:0/2468721777 wait complete. 2024-03-09T00:09:55.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:55 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/500735749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:55.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:55 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/500735749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:55.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:55 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/500735749' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:09:55.838 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:09:55.838 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:09:55.839 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:09:55.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 -- 172.21.15.73:0/1512275422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 msgr2=0x7f31281007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 --2- 172.21.15.73:0/1512275422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f31281007b0 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 crypto rx=0x7f3118009900 tx=0x7f311802bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 -- 172.21.15.73:0/1512275422 shutdown_connections 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 --2- 172.21.15.73:0/1512275422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281054a0 0x7f312810d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 --2- 172.21.15.73:0/1512275422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3128100d80 0x7f3128104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 --2- 172.21.15.73:0/1512275422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f31281007b0 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 -- 172.21.15.73:0/1512275422 >> 172.21.15.73:0/1512275422 conn(0x7f31280772f0 msgr2=0x7f3128077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:55.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 -- 172.21.15.73:0/1512275422 shutdown_connections 2024-03-09T00:09:55.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.960+0000 7f312e2ea640 1 -- 172.21.15.73:0/1512275422 wait complete. 2024-03-09T00:09:55.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 Processor -- start 2024-03-09T00:09:55.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 -- start start 2024-03-09T00:09:55.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31281054a0 0x7f3128198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281994d0 0x7f31281a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3128110560 con 0x7f31281003d0 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281103e0 con 0x7f31281054a0 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.961+0000 7f312e2ea640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281106e0 con 0x7f31281994d0 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56964/0 (socket says 172.21.15.73:56964) 2024-03-09T00:09:55.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 -- 172.21.15.73:0/2001211287 learned_addr learned my addr 172.21.15.73:0/2001211287 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f312c887640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281994d0 0x7f31281a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f31277fe640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31281054a0 0x7f3128198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 -- 172.21.15.73:0/2001211287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281994d0 msgr2=0x7f31281a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281994d0 0x7f31281a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 -- 172.21.15.73:0/2001211287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31281054a0 msgr2=0x7f3128198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31281054a0 0x7f3128198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 -- 172.21.15.73:0/2001211287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31281a4280 con 0x7f31281003d0 2024-03-09T00:09:55.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f3127fff640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3118006010 tx=0x7f311802cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f31257fa640 1 -- 172.21.15.73:0/2001211287 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118041070 con 0x7f31281003d0 2024-03-09T00:09:55.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31281a4510 con 0x7f31281003d0 2024-03-09T00:09:55.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f31257fa640 1 -- 172.21.15.73:0/2001211287 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3118004450 con 0x7f31281003d0 2024-03-09T00:09:55.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f31257fa640 1 -- 172.21.15.73:0/2001211287 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3118034440 con 0x7f31281003d0 2024-03-09T00:09:55.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.962+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31281a4a00 con 0x7f31281003d0 2024-03-09T00:09:55.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f31257fa640 1 -- 172.21.15.73:0/2001211287 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3118044070 con 0x7f31281003d0 2024-03-09T00:09:55.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 Processor -- start 2024-03-09T00:09:55.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 -- start start 2024-03-09T00:09:55.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea0d80 0x557503ea1160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f31257fa640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 0x7f31100a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 0x557503ea9d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503e04b40 con 0x557503eaa280 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503dadd80 con 0x557503ea0d80 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3130fa7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503e00be0 con 0x557503ea16a0 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f3127fff640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 0x7f31100a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56968/0 (socket says 172.21.15.73:56968) 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f312c887640 1 -- 172.21.15.73:0/478343193 learned_addr learned my addr 172.21.15.73:0/478343193 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f312c887640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea0d80 0x557503ea1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f31257fa640 1 -- 172.21.15.73:0/2001211287 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f31180f0e20 con 0x7f31281003d0 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.965+0000 7f31277fe640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 0x557503ea9d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 -- 172.21.15.73:0/478343193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 msgr2=0x557503ea9d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 0x557503ea9d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 -- 172.21.15.73:0/478343193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea0d80 msgr2=0x557503ea1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea0d80 0x557503ea1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 -- 172.21.15.73:0/478343193 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557503be17e0 con 0x557503eaa280 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f31277fe640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 0x557503ea9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f3127fff640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 0x7f31100a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f31180bd190 tx=0x7f31180dd040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f312c887640 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3120013a80 tx=0x7f312000a590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=77bb8f6b969d3aa2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f310e7fc640 1 -- 172.21.15.73:0/478343193 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120004b20 con 0x557503eaa280 2024-03-09T00:09:55.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f310e7fc640 1 -- 172.21.15.73:0/478343193 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312000ee40 con 0x557503eaa280 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.966+0000 7f310e7fc640 1 -- 172.21.15.73:0/478343193 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31200056a0 con 0x557503eaa280 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- 172.21.15.73:0/478343193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 msgr2=0x557503eac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 crypto rx=0x7f3120013a80 tx=0x7f312000a590 comp rx=0 tx=0).stop 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- 172.21.15.73:0/478343193 shutdown_connections 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eac670 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea16a0 0x557503ea9d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- 172.21.15.73:0/478343193 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea0d80 0x557503ea1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- 172.21.15.73:0/478343193 >> 172.21.15.73:0/478343193 conn(0x557503e9b4b0 msgr2=0x557503e9b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- 172.21.15.73:0/478343193 shutdown_connections 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- 172.21.15.73:0/478343193 wait complete. 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 -- start start 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 Processor -- start 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.967+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ea7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503ea77a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503ead070 con 0x557503ea0d80 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503dc11c0 con 0x557503ea16a0 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x557503ead1f0 con 0x557503eaa280 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 10 client.0 _ll_register_callbacks cb 0x557503db07a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56976/0 (socket says 172.21.15.73:56976) 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 -- 172.21.15.73:0/1430477652 learned_addr learned my addr 172.21.15.73:0/1430477652 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:55.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31277fe640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ea7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 -- 172.21.15.73:0/1430477652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 msgr2=0x557503ea77a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503ea77a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 -- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 msgr2=0x557503ea7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ea7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 -- 172.21.15.73:0/1430477652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557503c41f70 con 0x557503ea0d80 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31277fe640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ea7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:09:55.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f312c887640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f312000afd0 tx=0x7f312000adc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1772ac1675361f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31077fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312000eb60 con 0x557503ea0d80 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f31200053e0 con 0x557503ea0d80 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f312000ee40 con 0x557503ea0d80 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3120030050 con 0x557503ea0d80 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 10 client.16233 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.968+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x557503ad9170 con 0x557503ea0d80 2024-03-09T00:09:55.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312000ee40 con 0x557503ea0d80 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f312001e930 con 0x557503ea0d80 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 10 client.16233 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 10 client.16233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 20 client.16233 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503eba330 0x557503ebc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 4 client.16233 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557503dda960 con 0x557503eba330 2024-03-09T00:09:55.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f31277fe640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503eba330 0x557503ebc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f31277fe640 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503eba330 0x557503ebc720 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc36d440b67a102a server_cookie=dd9c56018f375663 in_seq=0 out_seq=0 2024-03-09T00:09:55.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.969+0000 7f31077fe640 10 client.16233 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:55.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f31077fe640 1 -- 172.21.15.73:0/1430477652 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x557503dda960 con 0x557503eba330 2024-03-09T00:09:55.980 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f31077fe640 10 client.16233 handle_command_reply: tid=0 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 client.16233 shutdown 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 msgr2=0x557503ea9cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 crypto rx=0x7f312000afd0 tx=0x7f312000adc0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 shutdown_connections 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503eba330 0x557503ebc720 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503ea77a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ea7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.981 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 --2- 172.21.15.73:0/1430477652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503ea9cd0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f31077fe640 0 client.16233 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 >> 172.21.15.73:0/1430477652 conn(0x557503e9b4b0 msgr2=0x557503e9d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 shutdown_connections 2024-03-09T00:09:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 1 -- 172.21.15.73:0/1430477652 wait complete. 2024-03-09T00:09:55.982 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.970+0000 7f3130fa7740 20 client.16233 trim_cache size 0 max 16384 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.985+0000 7f3130fa7740 1 Processor -- start 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.985+0000 7f3130fa7740 1 -- start start 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.985+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea0d80 0x557503ea1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.985+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ebc720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.985+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f3130fa7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281ac790 con 0x557503eaa280 2024-03-09T00:09:55.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f3130fa7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281aed50 con 0x557503ea16a0 2024-03-09T00:09:55.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f3130fa7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281acd90 con 0x557503ea0d80 2024-03-09T00:09:55.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:56984/0 (socket says 172.21.15.73:56984) 2024-03-09T00:09:55.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 -- 172.21.15.73:0/760698852 learned_addr learned my addr 172.21.15.73:0/760698852 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f31277fe640 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ebc720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea0d80 0x557503ea1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 -- 172.21.15.73:0/760698852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea0d80 msgr2=0x557503ea1a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea0d80 0x557503ea1a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 -- 172.21.15.73:0/760698852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 msgr2=0x557503ebc720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ebc720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 -- 172.21.15.73:0/760698852 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557503ea14d0 con 0x557503eaa280 2024-03-09T00:09:55.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f312c887640 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f31200052b0 tx=0x7f3120005c80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a5578ca9984c1c49 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f310dffb640 1 -- 172.21.15.73:0/760698852 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3120004f90 con 0x557503eaa280 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f310dffb640 1 -- 172.21.15.73:0/760698852 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312000ee40 con 0x557503eaa280 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.986+0000 7f310dffb640 1 -- 172.21.15.73:0/760698852 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312001e6b0 con 0x557503eaa280 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- 172.21.15.73:0/760698852 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 msgr2=0x557503eb1460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 crypto rx=0x7f31200052b0 tx=0x7f3120005c80 comp rx=0 tx=0).stop 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- 172.21.15.73:0/760698852 shutdown_connections 2024-03-09T00:09:55.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503eaa280 0x557503eb1460 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503ebc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 --2- 172.21.15.73:0/760698852 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503ea0d80 0x557503ea1a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- 172.21.15.73:0/760698852 >> 172.21.15.73:0/760698852 conn(0x557503e9b4b0 msgr2=0x557503e9c6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- 172.21.15.73:0/760698852 shutdown_connections 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- 172.21.15.73:0/760698852 wait complete. 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 -- start start 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 Processor -- start 2024-03-09T00:09:55.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.987+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503eaec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503f0e9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281ad3d0 con 0x557503ea0d80 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281ae6f0 con 0x557503ea16a0 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31281ada30 con 0x557503eaa280 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 10 client.0 _ll_register_callbacks cb 0x7f312000ab50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:55.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57000/0 (socket says 172.21.15.73:57000) 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 -- 172.21.15.73:0/3538282652 learned_addr learned my addr 172.21.15.73:0/3538282652 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31277fe640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503eaec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 -- 172.21.15.73:0/3538282652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 msgr2=0x557503f0e9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503f0e9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 -- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 msgr2=0x557503eaec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:55.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503eaec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 -- 172.21.15.73:0/3538282652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557503f0ef90 con 0x557503ea0d80 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f312c887640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f312000a8e0 tx=0x7f312000aa30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e476a8014248cfc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31077fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f312003b040 con 0x557503ea0d80 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f31200054f0 con 0x557503ea0d80 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f312000ee40 con 0x557503ea0d80 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f312000e960 con 0x557503ea0d80 2024-03-09T00:09:55.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 10 client.16239 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x557503f16380 con 0x557503ea0d80 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.988+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f312000ee40 con 0x557503ea0d80 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f312001e9e0 con 0x557503ea0d80 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 10 client.16239 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 20 client.16239 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503f16980 0x557503f18da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:55.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 4 client.16239 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x557503dc6080 con 0x557503f16980 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f31277fe640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503f16980 0x557503f18da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f31277fe640 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503f16980 0x557503f18da0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=27f9f55b76cf6b41 server_cookie=29c5e1395573f39f in_seq=0 out_seq=0 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.989+0000 7f31077fe640 10 client.16239 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f31077fe640 1 -- 172.21.15.73:0/3538282652 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x557503dc6080 con 0x557503f16980 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f31077fe640 10 client.16239 handle_command_reply: tid=0 2024-03-09T00:09:55.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 client.16239 shutdown 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 msgr2=0x557503eae710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 secure :-1 s=READY pgs=699 cs=0 l=1 rev1=1 crypto rx=0x7f312000a8e0 tx=0x7f312000aa30 comp rx=0 tx=0).stop 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 shutdown_connections 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x557503f16980 0x557503f18da0 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x557503eaa280 0x557503f0e9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x557503ea16a0 0x557503eaec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:55.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 --2- 172.21.15.73:0/3538282652 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x557503ea0d80 0x557503eae710 unknown :-1 s=CLOSED pgs=699 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f31077fe640 0 client.16239 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:56.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 >> 172.21.15.73:0/3538282652 conn(0x557503e9b4b0 msgr2=0x557503e9a300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 shutdown_connections 2024-03-09T00:09:56.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 1 -- 172.21.15.73:0/3538282652 wait complete. 2024-03-09T00:09:56.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.991+0000 7f3130fa7740 20 client.16239 trim_cache size 0 max 16384 2024-03-09T00:09:56.001 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:56.001 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:09:56.001 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "56ca383b-a60e-4efa-8187-52e0138fd2f1", 2024-03-09T00:09:56.001 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:09:56.001 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:56.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 msgr2=0x7f31100a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 0x7f31100a4e30 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f31180bd190 tx=0x7f31180dd040 comp rx=0 tx=0).stop 2024-03-09T00:09:56.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 msgr2=0x7f3128198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 crypto rx=0x7f3118006010 tx=0x7f311802cc70 comp rx=0 tx=0).stop 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 shutdown_connections 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f31281994d0 0x7f31281a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f31281054a0 0x7f3128198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f31100a2970 0x7f31100a4e30 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 --2- 172.21.15.73:0/2001211287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f31281003d0 0x7f3128198a00 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 >> 172.21.15.73:0/2001211287 conn(0x7f31280772f0 msgr2=0x7f31280fe3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 shutdown_connections 2024-03-09T00:09:56.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:55.992+0000 7f312e2ea640 1 -- 172.21.15.73:0/2001211287 wait complete. 2024-03-09T00:09:56.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:09:56.005 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 -- 172.21.15.73:0/688227578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa81014f0 msgr2=0x7f0aa81018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/688227578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa81014f0 0x7f0aa81018d0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009900 tx=0x7f0aa402bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 -- 172.21.15.73:0/688227578 shutdown_connections 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/688227578 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 0x7f0aa810e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/688227578 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa8101ea0 0x7f0aa8105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/688227578 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa81014f0 0x7f0aa81018d0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.119+0000 7f0aaedf1640 1 -- 172.21.15.73:0/688227578 >> 172.21.15.73:0/688227578 conn(0x7f0aa80fcd90 msgr2=0x7f0aa80ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- 172.21.15.73:0/688227578 shutdown_connections 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- 172.21.15.73:0/688227578 wait complete. 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 Processor -- start 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- start start 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa81014f0 0x7f0aa819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 0x7f0aa819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa8111680 con 0x7f0aa8197a60 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa8111500 con 0x7f0aa81014f0 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aaedf1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa8111800 con 0x7f0aa81065c0 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.120+0000 7f0aad38e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa81014f0 0x7f0aa819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57010/0 (socket says 172.21.15.73:57010) 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 -- 172.21.15.73:0/562950262 learned_addr learned my addr 172.21.15.73:0/562950262 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 0x7f0aa819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 -- 172.21.15.73:0/562950262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 msgr2=0x7f0aa819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 0x7f0aa819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 -- 172.21.15.73:0/562950262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa81014f0 msgr2=0x7f0aa819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa81014f0 0x7f0aa819d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 -- 172.21.15.73:0/562950262 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa8198580 con 0x7f0aa8197a60 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aad38e640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000eb10 tx=0x7f0aa000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0a9dffb640 1 -- 172.21.15.73:0/562950262 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0004b20 con 0x7f0aa8197a60 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa8198870 con 0x7f0aa8197a60 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0a9dffb640 1 -- 172.21.15.73:0/562950262 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0aa0004cc0 con 0x7f0aa8197a60 2024-03-09T00:09:56.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0a9dffb640 1 -- 172.21.15.73:0/562950262 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa0010760 con 0x7f0aa8197a60 2024-03-09T00:09:56.121 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.121+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa81a4a70 con 0x7f0aa8197a60 2024-03-09T00:09:56.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.124+0000 7f0a9dffb640 1 -- 172.21.15.73:0/562950262 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0aa0005000 con 0x7f0aa8197a60 2024-03-09T00:09:56.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.124+0000 7f0a9dffb640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 0x7f0a900a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.124+0000 7f0ab19d0740 1 Processor -- start 2024-03-09T00:09:56.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.124+0000 7f0ab19d0740 1 -- start start 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 0x7f0a900a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9dffb640 1 -- 172.21.15.73:0/562950262 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0aa00cf830 con 0x7f0aa8197a60 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadcd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1fad24e00 con 0x55c1fadcac60 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1faccdb10 con 0x55c1fadc1760 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1fad20ea0 con 0x55c1fadc2080 2024-03-09T00:09:56.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aad38e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadcd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:38528/0 (socket says 172.21.15.73:38528) 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:34732/0 (socket says 172.21.15.73:34732) 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aacb8d640 1 -- 172.21.15.73:0/2192925603 learned_addr learned my addr 172.21.15.73:0/2192925603 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 0x7f0a900a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009230 tx=0x7f0aa402c670 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 -- 172.21.15.73:0/2192925603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 msgr2=0x55c1fadc1b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc1b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 -- 172.21.15.73:0/2192925603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 msgr2=0x55c1fadcd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadcd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.125+0000 7f0a9ffff640 1 -- 172.21.15.73:0/2192925603 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1fab037d0 con 0x55c1fadc2080 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.126+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8199200 tx=0x7f0a98011440 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1794d8a6fbe46f1d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.126+0000 7f0a8effd640 1 -- 172.21.15.73:0/2192925603 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98004050 con 0x55c1fadc2080 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.126+0000 7f0a8effd640 1 -- 172.21.15.73:0/2192925603 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a98005500 con 0x55c1fadc2080 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.126+0000 7f0a8effd640 1 -- 172.21.15.73:0/2192925603 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98004ce0 con 0x55c1fadc2080 2024-03-09T00:09:56.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2192925603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 msgr2=0x55c1fadca720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f0aa8199200 tx=0x7f0a98011440 comp rx=0 tx=0).stop 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2192925603 shutdown_connections 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadcd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca720 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2192925603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc1b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2192925603 >> 172.21.15.73:0/2192925603 conn(0x55c1fadbbe90 msgr2=0x55c1fadbc270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2192925603 shutdown_connections 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2192925603 wait complete. 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 -- start start 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.127+0000 7f0ab19d0740 1 Processor -- start 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 0x55c1fadca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadcac60 0x55c1fadc7be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1fadcda50 con 0x55c1fadc2080 2024-03-09T00:09:56.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1face10a0 con 0x55c1fadcac60 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1fadcdbd0 con 0x55c1fadc1760 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 10 client.0 _ll_register_callbacks cb 0x55c1face0a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57030/0 (socket says 172.21.15.73:57030) 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 -- 172.21.15.73:0/629946901 learned_addr learned my addr 172.21.15.73:0/629946901 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0aad38e640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadcac60 0x55c1fadc7be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 0x55c1fadca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 -- 172.21.15.73:0/629946901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 msgr2=0x55c1fadca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 0x55c1fadca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 -- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadcac60 msgr2=0x55c1fadc7be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadcac60 0x55c1fadc7be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 -- 172.21.15.73:0/629946901 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1faa393f0 con 0x55c1fadc2080 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 0x55c1fadca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004880 tx=0x7f0a980045a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97deb8845ecf8b20 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a87fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a98003a00 con 0x55c1fadc2080 2024-03-09T00:09:56.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0a98005500 con 0x55c1fadc2080 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.128+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0a98003ba0 con 0x55c1fadc2080 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0a98032050 con 0x55c1fadc2080 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 10 client.16251 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c1fa9f90f0 con 0x55c1fadc2080 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a98005500 con 0x55c1fadc2080 2024-03-09T00:09:56.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0a98018070 con 0x55c1fadc2080 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 10 client.16251 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 10 client.16251 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 20 client.16251 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fadda540 0x55c1faddc930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 4 client.16251 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c1fad32340 con 0x55c1fadda540 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.129+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fadda540 0x55c1faddc930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.130+0000 7f0a87fff640 10 client.16251 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:56.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.130+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fadda540 0x55c1faddc930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53ea4f3ba847b65f server_cookie=7c153c62bc0be524 in_seq=0 out_seq=0 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0a87fff640 1 -- 172.21.15.73:0/629946901 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c1fad32340 con 0x55c1fadda540 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0a87fff640 10 client.16251 handle_command_reply: tid=0 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 client.16251 shutdown 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 msgr2=0x55c1fadc76a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 crypto rx=0x7f0a98004880 tx=0x7f0a980045a0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 shutdown_connections 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fadda540 0x55c1faddc930 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadcac60 0x55c1fadc7be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc2080 0x55c1fadc76a0 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0a87fff640 0 client.16251 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/629946901 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc1760 0x55c1fadca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 >> 172.21.15.73:0/629946901 conn(0x55c1fadbbe90 msgr2=0x55c1fadbdbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 shutdown_connections 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 1 -- 172.21.15.73:0/629946901 wait complete. 2024-03-09T00:09:56.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.131+0000 7f0ab19d0740 20 client.16251 trim_cache size 0 max 16384 2024-03-09T00:09:56.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 Processor -- start 2024-03-09T00:09:56.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 -- start start 2024-03-09T00:09:56.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc9dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81aa5e0 con 0x55c1fadcac60 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81acba0 con 0x55c1fadc1760 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81aabe0 con 0x55c1fadc2080 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0aad38e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0aad38e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57046/0 (socket says 172.21.15.73:57046) 2024-03-09T00:09:56.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0aad38e640 1 -- 172.21.15.73:0/586259839 learned_addr learned my addr 172.21.15.73:0/586259839 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc9dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.148+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 -- 172.21.15.73:0/586259839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 msgr2=0x55c1fadca310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 -- 172.21.15.73:0/586259839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 msgr2=0x55c1fadc9dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc9dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 -- 172.21.15.73:0/586259839 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1fade1a20 con 0x55c1fadcac60 2024-03-09T00:09:56.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0aad38e640 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0097f60 tx=0x7f0aa0086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a42efb7c80481d17 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0a8e7fc640 1 -- 172.21.15.73:0/586259839 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa009bb00 con 0x55c1fadcac60 2024-03-09T00:09:56.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0a8e7fc640 1 -- 172.21.15.73:0/586259839 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0aa009fc50 con 0x55c1fadcac60 2024-03-09T00:09:56.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.149+0000 7f0a8e7fc640 1 -- 172.21.15.73:0/586259839 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa00cf3c0 con 0x55c1fadcac60 2024-03-09T00:09:56.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- 172.21.15.73:0/586259839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 msgr2=0x55c1fadc5f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 crypto rx=0x7f0aa0097f60 tx=0x7f0aa0086040 comp rx=0 tx=0).stop 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- 172.21.15.73:0/586259839 shutdown_connections 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadcac60 0x55c1fadc5f90 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadc2080 0x55c1fadca310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/586259839 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc1760 0x55c1fadc9dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- 172.21.15.73:0/586259839 >> 172.21.15.73:0/586259839 conn(0x55c1fadbbe90 msgr2=0x55c1fadc7660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- 172.21.15.73:0/586259839 shutdown_connections 2024-03-09T00:09:56.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- 172.21.15.73:0/586259839 wait complete. 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 -- start start 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.150+0000 7f0ab19d0740 1 Processor -- start 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc2080 0x55c1fadc3d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadcac60 0x55c1fadc4280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81ab220 con 0x55c1fadc1760 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81ac540 con 0x55c1fadc2080 2024-03-09T00:09:56.153 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa81ab880 con 0x55c1fadcac60 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0ab19d0740 10 client.0 _ll_register_callbacks cb 0x7f0aa4044070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:57048/0 (socket says 172.21.15.73:57048) 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 -- 172.21.15.73:0/2219708242 learned_addr learned my addr 172.21.15.73:0/2219708242 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc2080 0x55c1fadc3d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aad38e640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadcac60 0x55c1fadc4280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 -- 172.21.15.73:0/2219708242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadcac60 msgr2=0x55c1fadc4280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadcac60 0x55c1fadc4280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 -- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc2080 msgr2=0x55c1fadc3d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc2080 0x55c1fadc3d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.151+0000 7f0aacb8d640 1 -- 172.21.15.73:0/2219708242 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1fadc47c0 con 0x55c1fadc1760 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0aacb8d640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000c00 tx=0x7f0aa402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6723132642f1ad05 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0aa404d040 con 0x55c1fadc1760 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0aa4004300 con 0x55c1fadc1760 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0aa009fc50 con 0x55c1fadc1760 2024-03-09T00:09:56.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0aa4067050 con 0x55c1fadc1760 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 10 client.16257 fetch_fsmap learned FSMap version 20 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c1fae31600 con 0x55c1fadc1760 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0aa009fc50 con 0x55c1fadc1760 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0aa4056580 con 0x55c1fadc1760 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 10 client.16257 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 10 client.16257 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:09:56.156 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 20 client.16257 populate_metadata read hostname 'smithi073' 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fae31d00 0x55c1fae34120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 4 client.16257 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.152+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c1fad2c690 con 0x55c1fae31d00 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fae31d00 0x55c1fae34120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0a87fff640 10 client.16257 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0a9ffff640 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fae31d00 0x55c1fae34120 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e36d7072be30d404 server_cookie=ce4df40d8671aab in_seq=0 out_seq=0 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0a87fff640 1 -- 172.21.15.73:0/2219708242 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55c1fad2c690 con 0x55c1fae31d00 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0a87fff640 10 client.16257 handle_command_reply: tid=0 2024-03-09T00:09:56.157 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0ab19d0740 1 client.16257 shutdown 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 msgr2=0x55c1fadd2000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.153+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4000c00 tx=0x7f0aa402bc30 comp rx=0 tx=0).stop 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 shutdown_connections 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1fae31d00 0x55c1fae34120 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1fadcac60 0x55c1fadc4280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1fadc2080 0x55c1fadc3d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 --2- 172.21.15.73:0/2219708242 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1fadc1760 0x55c1fadd2000 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.158 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 >> 172.21.15.73:0/2219708242 conn(0x55c1fadbbe90 msgr2=0x55c1fadc5ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0a87fff640 0 client.16257 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 shutdown_connections 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 1 -- 172.21.15.73:0/2219708242 wait complete. 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0ab19d0740 20 client.16257 trim_cache size 0 max 16384 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:09:56.159 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2398 inodes in the stack)", 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: "56ca383b-a60e-4efa-8187-52e0138fd2f1": { 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "56ca383b-a60e-4efa-8187-52e0138fd2f1", 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:09:56.160 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:09:56.161 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 msgr2=0x7f0a900a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 0x7f0a900a4e30 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0aa4009230 tx=0x7f0aa402c670 comp rx=0 tx=0).stop 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 msgr2=0x7f0aa8197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 crypto rx=0x7f0aa000eb10 tx=0x7f0aa000c2d0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 shutdown_connections 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0a900a2970 0x7f0a900a4e30 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0aa8197a60 0x7f0aa8197f10 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0aa81065c0 0x7f0aa819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.162 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 --2- 172.21.15.73:0/562950262 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0aa81014f0 0x7f0aa819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:09:56.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 >> 172.21.15.73:0/562950262 conn(0x7f0aa80fcd90 msgr2=0x7f0aa80fe160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:09:56.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 shutdown_connections 2024-03-09T00:09:56.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:09:56.154+0000 7f0aaedf1640 1 -- 172.21.15.73:0/562950262 wait complete. 2024-03-09T00:09:56.163 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:56 smithi073 ceph-mon[32309]: pgmap v95: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:56.164 INFO:tasks.cephfs.filesystem:scrub status for tag:56ca383b-a60e-4efa-8187-52e0138fd2f1 - {'path': '/', 'tag': '56ca383b-a60e-4efa-8187-52e0138fd2f1', 'options': 'recursive,force'} 2024-03-09T00:09:56.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:56 smithi179 ceph-mon[36637]: pgmap v95: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:56.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:56 smithi102 ceph-mon[36106]: pgmap v95: 129 pgs: 129 active+clean; 35 MiB data, 701 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:09:57.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:57 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:09:57.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:57 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:57.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:57 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:09:57.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:57 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:57 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:09:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:57 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:57 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:09:57.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:57 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:57.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:57 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:09:57.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:57 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:57.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:57 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:09:57.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:57 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:09:58.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:58 smithi073 ceph-mon[32309]: pgmap v96: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 155 op/s 2024-03-09T00:09:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:58 smithi102 ceph-mon[36106]: pgmap v96: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 155 op/s 2024-03-09T00:09:58.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:58 smithi179 ceph-mon[36637]: pgmap v96: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 3.7 KiB/s wr, 155 op/s 2024-03-09T00:09:59.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:09:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:09:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:09:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:09:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:09:59 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:09:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:09:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:09:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:09:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:09:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:09:59 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:09:59.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:09:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:09:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:09:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:09:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:09:59 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:10:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:00 smithi073 ceph-mon[32309]: pgmap v97: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:00.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:00 smithi073 ceph-mon[32309]: overall HEALTH_OK 2024-03-09T00:10:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:00 smithi102 ceph-mon[36106]: pgmap v97: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:00.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:00 smithi102 ceph-mon[36106]: overall HEALTH_OK 2024-03-09T00:10:00.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:00 smithi179 ceph-mon[36637]: pgmap v97: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:00.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:00 smithi179 ceph-mon[36637]: overall HEALTH_OK 2024-03-09T00:10:02.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:02 smithi073 ceph-mon[32309]: pgmap v98: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:02.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:02 smithi102 ceph-mon[36106]: pgmap v98: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:02.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:02 smithi179 ceph-mon[36637]: pgmap v98: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 291 KiB/s rd, 3.7 KiB/s wr, 288 op/s 2024-03-09T00:10:04.331 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:04 smithi179 ceph-mon[36637]: pgmap v99: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 26 KiB/s wr, 304 op/s 2024-03-09T00:10:04.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:04 smithi073 ceph-mon[32309]: pgmap v99: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 26 KiB/s wr, 304 op/s 2024-03-09T00:10:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:04 smithi102 ceph-mon[36106]: pgmap v99: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 26 KiB/s wr, 304 op/s 2024-03-09T00:10:06.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:06 smithi073 ceph-mon[32309]: pgmap v100: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 26 KiB/s wr, 303 op/s 2024-03-09T00:10:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:06 smithi102 ceph-mon[36106]: pgmap v100: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 26 KiB/s wr, 303 op/s 2024-03-09T00:10:06.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:06 smithi179 ceph-mon[36637]: pgmap v100: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 26 KiB/s wr, 303 op/s 2024-03-09T00:10:08.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:08 smithi073 ceph-mon[32309]: pgmap v101: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 29 KiB/s wr, 304 op/s 2024-03-09T00:10:08.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:08 smithi102 ceph-mon[36106]: pgmap v101: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 29 KiB/s wr, 304 op/s 2024-03-09T00:10:08.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:08 smithi179 ceph-mon[36637]: pgmap v101: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 29 KiB/s wr, 304 op/s 2024-03-09T00:10:10.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:10 smithi073 ceph-mon[32309]: pgmap v102: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 KiB/s wr, 151 op/s 2024-03-09T00:10:10.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:10 smithi102 ceph-mon[36106]: pgmap v102: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 KiB/s wr, 151 op/s 2024-03-09T00:10:10.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:10 smithi179 ceph-mon[36637]: pgmap v102: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 25 KiB/s wr, 151 op/s 2024-03-09T00:10:12.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:12 smithi073 ceph-mon[32309]: pgmap v103: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-03-09T00:10:12.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:12 smithi102 ceph-mon[36106]: pgmap v103: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-03-09T00:10:12.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:12 smithi179 ceph-mon[36637]: pgmap v103: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 25 KiB/s wr, 16 op/s 2024-03-09T00:10:14.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:14 smithi073 ceph-mon[32309]: pgmap v104: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-03-09T00:10:14.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:14 smithi102 ceph-mon[36106]: pgmap v104: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-03-09T00:10:14.581 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:14 smithi179 ceph-mon[36637]: pgmap v104: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 25 KiB/s wr, 17 op/s 2024-03-09T00:10:16.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:16 smithi179 ceph-mon[36637]: pgmap v105: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 KiB/s wr, 2 op/s 2024-03-09T00:10:16.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:16 smithi073 ceph-mon[32309]: pgmap v105: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 KiB/s wr, 2 op/s 2024-03-09T00:10:16.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:16 smithi102 ceph-mon[36106]: pgmap v105: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 3.3 KiB/s wr, 2 op/s 2024-03-09T00:10:18.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:18 smithi073 ceph-mon[32309]: pgmap v106: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-03-09T00:10:18.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:18 smithi102 ceph-mon[36106]: pgmap v106: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-03-09T00:10:18.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:18 smithi179 ceph-mon[36637]: pgmap v106: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 4.9 KiB/s wr, 3 op/s 2024-03-09T00:10:20.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:20 smithi073 ceph-mon[32309]: pgmap v107: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:20.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:20 smithi102 ceph-mon[36106]: pgmap v107: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:20.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:20 smithi179 ceph-mon[36637]: pgmap v107: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:22.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:22 smithi073 ceph-mon[32309]: pgmap v108: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:22.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:22 smithi102 ceph-mon[36106]: pgmap v108: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:22.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:22 smithi179 ceph-mon[36637]: pgmap v108: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:24.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:24 smithi073 ceph-mon[32309]: pgmap v109: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:24.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:24 smithi102 ceph-mon[36106]: pgmap v109: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:24.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:24 smithi179 ceph-mon[36637]: pgmap v109: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:26.165 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:10:26.166 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:10:26.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 -- 172.21.15.73:0/3839438798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c1018e0 msgr2=0x7ff79c104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/3839438798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c1018e0 0x7ff79c104e80 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009900 tx=0x7ff78c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:10:26.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 -- 172.21.15.73:0/3839438798 shutdown_connections 2024-03-09T00:10:26.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/3839438798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 0x7ff79c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/3839438798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c1018e0 0x7ff79c104e80 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/3839438798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c100f30 0x7ff79c101310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 -- 172.21.15.73:0/3839438798 >> 172.21.15.73:0/3839438798 conn(0x7ff79c075530 msgr2=0x7ff79c075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 -- 172.21.15.73:0/3839438798 shutdown_connections 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.288+0000 7ff7a32f1640 1 -- 172.21.15.73:0/3839438798 wait complete. 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 Processor -- start 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 -- start start 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 0x7ff79c19ddd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 0x7ff79c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c110590 con 0x7ff79c100f30 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c110410 con 0x7ff79c1055b0 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a32f1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c110710 con 0x7ff79c197a40 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a108d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a108d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40828/0 (socket says 172.21.15.73:40828) 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.289+0000 7ff7a108d640 1 -- 172.21.15.73:0/1920256177 learned_addr learned my addr 172.21.15.73:0/1920256177 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a188e640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 0x7ff79c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a088c640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 0x7ff79c19ddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 -- 172.21.15.73:0/1920256177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 msgr2=0x7ff79c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 0x7ff79c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 -- 172.21.15.73:0/1920256177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 msgr2=0x7ff79c19ddd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 0x7ff79c19ddd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 -- 172.21.15.73:0/1920256177 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff79c198560 con 0x7ff79c100f30 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a188e640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 0x7ff79c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a108d640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff79800c970 tx=0x7ff79800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7927fc640 1 -- 172.21.15.73:0/1920256177 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff798007bb0 con 0x7ff79c100f30 2024-03-09T00:10:26.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7927fc640 1 -- 172.21.15.73:0/1920256177 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff798007d50 con 0x7ff79c100f30 2024-03-09T00:10:26.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff79c198850 con 0x7ff79c100f30 2024-03-09T00:10:26.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7927fc640 1 -- 172.21.15.73:0/1920256177 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79800f450 con 0x7ff79c100f30 2024-03-09T00:10:26.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.290+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff79c1a4a50 con 0x7ff79c100f30 2024-03-09T00:10:26.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.292+0000 7ff7927fc640 1 -- 172.21.15.73:0/1920256177 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff798005020 con 0x7ff79c100f30 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 Processor -- start 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 -- start start 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c32b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7927fc640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 0x7ff7880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b95e00 con 0x560b57c3bc60 2024-03-09T00:10:26.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b3eb10 con 0x560b57c33080 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b91ea0 con 0x560b57c32760 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a088c640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 0x7ff7880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40844/0 (socket says 172.21.15.73:40844) 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 -- 172.21.15.73:0/3501569030 learned_addr learned my addr 172.21.15.73:0/3501569030 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7927fc640 1 -- 172.21.15.73:0/1920256177 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff7980cf080 con 0x7ff79c100f30 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a108d640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c32b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 -- 172.21.15.73:0/3501569030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 msgr2=0x560b57c32b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c32b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 -- 172.21.15.73:0/3501569030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 msgr2=0x560b57c3b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a188e640 1 -- 172.21.15.73:0/3501569030 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b579747d0 con 0x560b57c3bc60 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.293+0000 7ff7a108d640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a088c640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 0x7ff7880a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009230 tx=0x7ff78c005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a188e640 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff794013840 tx=0x7ff7940099c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4adea64b93b3d50 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7877fe640 1 -- 172.21.15.73:0/3501569030 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794019030 con 0x560b57c3bc60 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7877fe640 1 -- 172.21.15.73:0/3501569030 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff794003e50 con 0x560b57c3bc60 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7877fe640 1 -- 172.21.15.73:0/3501569030 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff794011bf0 con 0x560b57c3bc60 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 -- 172.21.15.73:0/3501569030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 msgr2=0x560b57c3e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 crypto rx=0x7ff794013840 tx=0x7ff7940099c0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 -- 172.21.15.73:0/3501569030 shutdown_connections 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c3bc60 0x560b57c3e050 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/3501569030 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c32b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.294+0000 7ff7a5faa740 1 -- 172.21.15.73:0/3501569030 >> 172.21.15.73:0/3501569030 conn(0x560b57c2ce90 msgr2=0x560b57c2d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- 172.21.15.73:0/3501569030 shutdown_connections 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- 172.21.15.73:0/3501569030 wait complete. 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- start start 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 Processor -- start 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 0x560b57c3b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c38d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c3ea50 con 0x560b57c33080 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57b520a0 con 0x560b57c32760 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560b57c3ebd0 con 0x560b57c3bc60 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a5faa740 10 client.0 _ll_register_callbacks cb 0x560b57b51a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a088c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a088c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40860/0 (socket says 172.21.15.73:40860) 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.295+0000 7ff7a088c640 1 -- 172.21.15.73:0/2411210493 learned_addr learned my addr 172.21.15.73:0/2411210493 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a188e640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c38d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a108d640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 0x560b57c3b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 -- 172.21.15.73:0/2411210493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 msgr2=0x560b57c38d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c38d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 -- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 msgr2=0x560b57c3b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 0x560b57c3b450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 -- 172.21.15.73:0/2411210493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b578aa3f0 con 0x560b57c33080 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a188e640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c38d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a108d640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 0x560b57c3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:26.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a088c640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009900 tx=0x7ff78c009260 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9dbf36cd72536d41 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c04c070 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff794003e50 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff78c003cc0 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff78c058050 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a5faa740 10 client.16269 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560b5786a0f0 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.296+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff794003e50 con 0x560b57c33080 2024-03-09T00:10:26.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff78c047070 con 0x560b57c33080 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 10 client.16269 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 10 client.16269 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 20 client.16269 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57c4b600 0x560b57c4da30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 4 client.16269 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560b57ba3340 con 0x560b57c4b600 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a108d640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57c4b600 0x560b57c4da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff784ff9640 10 client.16269 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.297+0000 7ff7a108d640 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57c4b600 0x560b57c4da30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f037e03907e7c404 server_cookie=50394a7501aabdb5 in_seq=0 out_seq=0 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff784ff9640 1 -- 172.21.15.73:0/2411210493 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560b57ba3340 con 0x560b57c4b600 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff784ff9640 10 client.16269 handle_command_reply: tid=0 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 client.16269 shutdown 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 msgr2=0x560b57c387e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 secure :-1 s=READY pgs=708 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009900 tx=0x7ff78c009260 comp rx=0 tx=0).stop 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 shutdown_connections 2024-03-09T00:10:26.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57c4b600 0x560b57c4da30 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c38d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c387e0 unknown :-1 s=CLOSED pgs=708 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff784ff9640 0 client.16269 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/2411210493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c32760 0x560b57c3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 >> 172.21.15.73:0/2411210493 conn(0x560b57c2ce90 msgr2=0x560b57c2ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 shutdown_connections 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 1 -- 172.21.15.73:0/2411210493 wait complete. 2024-03-09T00:10:26.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.298+0000 7ff7a5faa740 20 client.16269 trim_cache size 0 max 16384 2024-03-09T00:10:26.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 Processor -- start 2024-03-09T00:10:26.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 -- start start 2024-03-09T00:10:26.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c35a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1ac780 con 0x560b57c32760 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1aed40 con 0x560b57c33080 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.313+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1acd80 con 0x560b57c3bc60 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40868/0 (socket says 172.21.15.73:40868) 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 -- 172.21.15.73:0/234569805 learned_addr learned my addr 172.21.15.73:0/234569805 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 -- 172.21.15.73:0/234569805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 msgr2=0x560b57c35a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c35a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 -- 172.21.15.73:0/234569805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 msgr2=0x560b57c3b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 -- 172.21.15.73:0/234569805 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560b57c32eb0 con 0x560b57c32760 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff7a108d640 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff798097d20 tx=0x7ff79809b120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=484e91e39b5ee707 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff786ffd640 1 -- 172.21.15.73:0/234569805 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff79801d040 con 0x560b57c32760 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff786ffd640 1 -- 172.21.15.73:0/234569805 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7980b9e50 con 0x560b57c32760 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.314+0000 7ff786ffd640 1 -- 172.21.15.73:0/234569805 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff7980cc6c0 con 0x560b57c32760 2024-03-09T00:10:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- 172.21.15.73:0/234569805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 msgr2=0x560b57c3af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 crypto rx=0x7ff798097d20 tx=0x7ff79809b120 comp rx=0 tx=0).stop 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- 172.21.15.73:0/234569805 shutdown_connections 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c3bc60 0x560b57c35a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c33080 0x560b57c3b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/234569805 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c32760 0x560b57c3af10 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- 172.21.15.73:0/234569805 >> 172.21.15.73:0/234569805 conn(0x560b57c2ce90 msgr2=0x560b57c387a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- 172.21.15.73:0/234569805 shutdown_connections 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- 172.21.15.73:0/234569805 wait complete. 2024-03-09T00:10:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- start start 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 Processor -- start 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c9bfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c3bc60 0x560b57c42120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1ad3c0 con 0x560b57c33080 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1ae6e0 con 0x560b57c3bc60 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.315+0000 7ff7a5faa740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff79c1ada20 con 0x560b57c32760 2024-03-09T00:10:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a5faa740 10 client.0 _ll_register_callbacks cb 0x7ff78c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40884/0 (socket says 172.21.15.73:40884) 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 -- 172.21.15.73:0/705761469 learned_addr learned my addr 172.21.15.73:0/705761469 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a188e640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c3bc60 0x560b57c42120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a108d640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c9bfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 -- 172.21.15.73:0/705761469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 msgr2=0x560b57c9bfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c9bfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 -- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c3bc60 msgr2=0x560b57c42120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c3bc60 0x560b57c42120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 -- 172.21.15.73:0/705761469 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560b57c42920 con 0x560b57c33080 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a088c640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff78c004ce0 tx=0x7ff78c0048f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998113d7ec79ad70 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff784ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff78c04c050 con 0x560b57c33080 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff7980b9e50 con 0x560b57c33080 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff78c003dd0 con 0x560b57c33080 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff78c06a050 con 0x560b57c33080 2024-03-09T00:10:26.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a5faa740 10 client.16275 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560b57ca36f0 con 0x560b57c33080 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.316+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff7980b9e50 con 0x560b57c33080 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff78c047040 con 0x560b57c33080 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 10 client.16275 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 10 client.16275 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 20 client.16275 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57ca3e40 0x560b57ca6260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.321 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 4 client.16275 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560b57b9d690 con 0x560b57ca3e40 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a108d640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57ca3e40 0x560b57ca6260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff784ff9640 10 client.16275 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.317+0000 7ff7a108d640 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57ca3e40 0x560b57ca6260 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6ba47847c71911d3 server_cookie=b8d779e9b1aec66f in_seq=0 out_seq=0 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff784ff9640 1 -- 172.21.15.73:0/705761469 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560b57b9d690 con 0x560b57ca3e40 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff784ff9640 10 client.16275 handle_command_reply: tid=0 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 client.16275 shutdown 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 msgr2=0x560b57c9c520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.322 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 crypto rx=0x7ff78c004ce0 tx=0x7ff78c0048f0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 shutdown_connections 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560b57ca3e40 0x560b57ca6260 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560b57c3bc60 0x560b57c42120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560b57c33080 0x560b57c9c520 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 --2- 172.21.15.73:0/705761469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560b57c32760 0x560b57c9bfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 >> 172.21.15.73:0/705761469 conn(0x560b57c2ce90 msgr2=0x560b57c33460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff784ff9640 0 client.16275 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 shutdown_connections 2024-03-09T00:10:26.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 1 -- 172.21.15.73:0/705761469 wait complete. 2024-03-09T00:10:26.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a5faa740 20 client.16275 trim_cache size 0 max 16384 2024-03-09T00:10:26.324 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:10:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:10:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:10:26.324 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:10:26.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 msgr2=0x7ff7880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 0x7ff7880a4e30 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff78c009230 tx=0x7ff78c005fd0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 msgr2=0x7ff79c19d890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.318+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 crypto rx=0x7ff79800c970 tx=0x7ff79800ce30 comp rx=0 tx=0).stop 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 shutdown_connections 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff79c197a40 0x7ff79c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff79c1055b0 0x7ff79c19ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff7880a2970 0x7ff7880a4e30 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 --2- 172.21.15.73:0/1920256177 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff79c100f30 0x7ff79c19d890 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 >> 172.21.15.73:0/1920256177 conn(0x7ff79c075530 msgr2=0x7ff79c0ffe90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 shutdown_connections 2024-03-09T00:10:26.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.319+0000 7ff7a32f1640 1 -- 172.21.15.73:0/1920256177 wait complete. 2024-03-09T00:10:26.328 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:10:26.328 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:10:26.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:26 smithi073 ceph-mon[32309]: pgmap v110: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:26.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 -- 172.21.15.73:0/1206995775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100310 msgr2=0x7feb901006f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 --2- 172.21.15.73:0/1206995775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100310 0x7feb901006f0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 crypto rx=0x7feb8c009930 tx=0x7feb8c02bcd0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 -- 172.21.15.73:0/1206995775 shutdown_connections 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 --2- 172.21.15.73:0/1206995775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 0x7feb90105bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 --2- 172.21.15.73:0/1206995775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100c30 0x7feb901050f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 --2- 172.21.15.73:0/1206995775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100310 0x7feb901006f0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.435+0000 7feb97329640 1 -- 172.21.15.73:0/1206995775 >> 172.21.15.73:0/1206995775 conn(0x7feb90076570 msgr2=0x7feb90076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- 172.21.15.73:0/1206995775 shutdown_connections 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- 172.21.15.73:0/1206995775 wait complete. 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 Processor -- start 2024-03-09T00:10:26.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- start start 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 0x7feb9019e1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 0x7feb901a2a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90117ca0 con 0x7feb90100c30 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90117b20 con 0x7feb90100310 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb97329640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb90117e20 con 0x7feb90105710 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.436+0000 7feb948c4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40896/0 (socket says 172.21.15.73:40896) 2024-03-09T00:10:26.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 -- 172.21.15.73:0/1886444816 learned_addr learned my addr 172.21.15.73:0/1886444816 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb958c6640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 0x7feb901a2a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb950c5640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 0x7feb9019e1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 -- 172.21.15.73:0/1886444816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 msgr2=0x7feb901a2a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 0x7feb901a2a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 -- 172.21.15.73:0/1886444816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 msgr2=0x7feb9019e1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 0x7feb9019e1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 -- 172.21.15.73:0/1886444816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb901a31b0 con 0x7feb90100c30 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb950c5640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 0x7feb9019e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb948c4640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7feb8000e9f0 tx=0x7feb8000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb8000cd80 con 0x7feb90100c30 2024-03-09T00:10:26.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb901a34a0 con 0x7feb90100c30 2024-03-09T00:10:26.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb80010910 con 0x7feb90100c30 2024-03-09T00:10:26.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb800188f0 con 0x7feb90100c30 2024-03-09T00:10:26.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.437+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb901aad30 con 0x7feb90100c30 2024-03-09T00:10:26.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.438+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9006a140 con 0x7feb90100c30 2024-03-09T00:10:26.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.439+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7feb80010ac0 con 0x7feb90100c30 2024-03-09T00:10:26.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.440+0000 7feb867fc640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 0x7feb7c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.440+0000 7feb950c5640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 0x7feb7c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.440+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7feb800cf650 con 0x7feb90100c30 2024-03-09T00:10:26.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.440+0000 7feb950c5640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 0x7feb7c0a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006010 tx=0x7feb8c009610 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.444+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feb8009c020 con 0x7feb90100c30 2024-03-09T00:10:26.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:26 smithi102 ceph-mon[36106]: pgmap v110: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:26.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:26 smithi179 ceph-mon[36637]: pgmap v110: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 1.6 KiB/s wr, 1 op/s 2024-03-09T00:10:26.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.655+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feb9019f510 con 0x7feb90100c30 2024-03-09T00:10:26.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.656+0000 7feb867fc640 1 -- 172.21.15.73:0/1886444816 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7feb8009b920 con 0x7feb90100c30 2024-03-09T00:10:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:10:26.651 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:10:26.651 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 msgr2=0x7feb7c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 0x7feb7c0a4e30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7feb8c006010 tx=0x7feb8c009610 comp rx=0 tx=0).stop 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 msgr2=0x7feb9019e700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 crypto rx=0x7feb8000e9f0 tx=0x7feb8000eeb0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 shutdown_connections 2024-03-09T00:10:26.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb90105710 0x7feb901a2a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb7c0a2970 0x7feb7c0a4e30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb90100c30 0x7feb9019e700 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 --2- 172.21.15.73:0/1886444816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb90100310 0x7feb9019e1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.657+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 >> 172.21.15.73:0/1886444816 conn(0x7feb90076570 msgr2=0x7feb901030c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.658+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 shutdown_connections 2024-03-09T00:10:26.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.658+0000 7feb97329640 1 -- 172.21.15.73:0/1886444816 wait complete. 2024-03-09T00:10:26.664 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:10:26.664 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:10:26.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 -- 172.21.15.73:0/3528377650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8100440 msgr2=0x7fbbc81008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3528377650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8100440 0x7fbbc81008c0 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009930 tx=0x7fbbbc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 -- 172.21.15.73:0/3528377650 shutdown_connections 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3528377650 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc810cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3528377650 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8100440 0x7fbbc81008c0 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3528377650 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8106310 0x7fbbc81066f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 -- 172.21.15.73:0/3528377650 >> 172.21.15.73:0/3528377650 conn(0x7fbbc8076f80 msgr2=0x7fbbc8077390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 -- 172.21.15.73:0/3528377650 shutdown_connections 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.772+0000 7fbbcd832640 1 -- 172.21.15.73:0/3528377650 wait complete. 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 Processor -- start 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 -- start start 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8100440 0x7fbbc8198950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc8198e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8110510 con 0x7fbbc8106310 2024-03-09T00:10:26.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8110390 con 0x7fbbc8100e00 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbcd832640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc8110690 con 0x7fbbc8100440 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40918/0 (socket says 172.21.15.73:40918) 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 -- 172.21.15.73:0/3168675850 learned_addr learned my addr 172.21.15.73:0/3168675850 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc8198e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8100440 0x7fbbc8198950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 -- 172.21.15.73:0/3168675850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8100440 msgr2=0x7fbbc8198950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8100440 0x7fbbc8198950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 -- 172.21.15.73:0/3168675850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 msgr2=0x7fbbc8198e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc8198e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 -- 172.21.15.73:0/3168675850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbc81a3f10 con 0x7fbbc8106310 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc8198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.773+0000 7fbbc77fe640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000eb10 tx=0x7fbbc000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.774+0000 7fbbcc830640 1 -- 172.21.15.73:0/3168675850 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0004b20 con 0x7fbbc8106310 2024-03-09T00:10:26.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.774+0000 7fbbcc830640 1 -- 172.21.15.73:0/3168675850 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbc0004cc0 con 0x7fbbc8106310 2024-03-09T00:10:26.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.774+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbc81a40e0 con 0x7fbbc8106310 2024-03-09T00:10:26.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.774+0000 7fbbcc830640 1 -- 172.21.15.73:0/3168675850 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbc0010760 con 0x7fbbc8106310 2024-03-09T00:10:26.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.774+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbc81a45e0 con 0x7fbbc8106310 2024-03-09T00:10:26.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbcc830640 1 -- 172.21.15.73:0/3168675850 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbbc0005020 con 0x7fbbc8106310 2024-03-09T00:10:26.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 Processor -- start 2024-03-09T00:10:26.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 -- start start 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 0x563bfad80580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8ba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bfad47360 con 0x563bfad80ac0 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bface3ec0 con 0x563bfad801a0 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbd040e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bfac8cd30 con 0x563bfad896a0 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.776+0000 7fbbc6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbcc830640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 0x7fbbb00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40932/0 (socket says 172.21.15.73:40932) 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/267587382 learned_addr learned my addr 172.21.15.73:0/267587382 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 0x563bfad80580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc77fe640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 0x7fbbb00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/267587382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 msgr2=0x563bfad8ba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbcc830640 1 -- 172.21.15.73:0/3168675850 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbbc00cf910 con 0x7fbbc8106310 2024-03-09T00:10:26.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8ba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/267587382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 msgr2=0x563bfad80580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 0x563bfad80580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/267587382 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bfac8bbc0 con 0x563bfad80ac0 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 0x563bfad80580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800cb40 tx=0x7fbbb800e350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=132600a9fb5656a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbc77fe640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 0x7fbbb00a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00a3df0 tx=0x7fbbc006a990 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbadffb640 1 -- 172.21.15.73:0/267587382 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8010430 con 0x563bfad80ac0 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbadffb640 1 -- 172.21.15.73:0/267587382 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb8010ab0 con 0x563bfad80ac0 2024-03-09T00:10:26.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.777+0000 7fbbadffb640 1 -- 172.21.15.73:0/267587382 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb800fb60 con 0x563bfad80ac0 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- 172.21.15.73:0/267587382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 msgr2=0x563bfad89160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800cb40 tx=0x7fbbb800e350 comp rx=0 tx=0).stop 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- 172.21.15.73:0/267587382 shutdown_connections 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8ba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad89160 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 --2- 172.21.15.73:0/267587382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad801a0 0x563bfad80580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- 172.21.15.73:0/267587382 >> 172.21.15.73:0/267587382 conn(0x563bfad7a8d0 msgr2=0x563bfad7acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- 172.21.15.73:0/267587382 shutdown_connections 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- 172.21.15.73:0/267587382 wait complete. 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 -- start start 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.778+0000 7fbbd040e740 1 Processor -- start 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad801a0 0x563bfad86680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad87100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bfac824a0 con 0x563bfad80ac0 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bfacdff60 con 0x563bfad896a0 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563bfad8c490 con 0x563bfad801a0 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 10 client.0 _ll_register_callbacks cb 0x563bfacddd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40938/0 (socket says 172.21.15.73:40938) 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/3178758539 learned_addr learned my addr 172.21.15.73:0/3178758539 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad87100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/3178758539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad801a0 msgr2=0x563bfad86680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad801a0 0x563bfad86680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 msgr2=0x563bfad87100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad87100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 -- 172.21.15.73:0/3178758539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bfa9f83f0 con 0x563bfad80ac0 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad87100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800efc0 tx=0x7fbbb8005740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c9a97bfc05732a30 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbba6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:26.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbb8010640 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbbb80107e0 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbb8010ab0 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbb8030050 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 10 client.16293 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.779+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563bfaac27d0 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbb8010ab0 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbbb8022070 con 0x563bfad80ac0 2024-03-09T00:10:26.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 10 client.16293 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 10 client.16293 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 20 client.16293 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfad99760 0x563bfad9bb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 4 client.16293 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563bfacf1400 con 0x563bfad99760 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfad99760 0x563bfad9bb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbba6ffd640 10 client.16293 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.780+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfad99760 0x563bfad9bb50 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7b7e534f57278a66 server_cookie=969d511b09fca8d7 in_seq=0 out_seq=0 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbba6ffd640 1 -- 172.21.15.73:0/3178758539 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563bfacf1400 con 0x563bfad99760 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbba6ffd640 10 client.16293 handle_command_reply: tid=0 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 client.16293 shutdown 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 msgr2=0x563bfad86bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 crypto rx=0x7fbbb800efc0 tx=0x7fbbb8005740 comp rx=0 tx=0).stop 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 shutdown_connections 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfad99760 0x563bfad9bb50 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad87100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad80ac0 0x563bfad86bc0 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbba6ffd640 0 client.16293 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 --2- 172.21.15.73:0/3178758539 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad801a0 0x563bfad86680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 >> 172.21.15.73:0/3178758539 conn(0x563bfad7a8d0 msgr2=0x563bfad7c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 shutdown_connections 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 1 -- 172.21.15.73:0/3178758539 wait complete. 2024-03-09T00:10:26.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.781+0000 7fbbd040e740 20 client.16293 trim_cache size 0 max 16384 2024-03-09T00:10:26.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 Processor -- start 2024-03-09T00:10:26.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 -- start start 2024-03-09T00:10:26.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad80ac0 0x563bfad838e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad83e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81ac7f0 con 0x563bfad801a0 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81aedb0 con 0x563bfad896a0 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbd040e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81acdf0 con 0x563bfad80ac0 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40952/0 (socket says 172.21.15.73:40952) 2024-03-09T00:10:26.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 -- 172.21.15.73:0/2139893409 learned_addr learned my addr 172.21.15.73:0/2139893409 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad80ac0 0x563bfad838e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 -- 172.21.15.73:0/2139893409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad80ac0 msgr2=0x563bfad838e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad80ac0 0x563bfad838e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 -- 172.21.15.73:0/2139893409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 msgr2=0x563bfad83e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad83e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.793+0000 7fbbc67fc640 1 -- 172.21.15.73:0/2139893409 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563bfad76190 con 0x563bfad801a0 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc036040 tx=0x7fbbbc0092d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=607466f9f2591bcd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbad7fa640 1 -- 172.21.15.73:0/2139893409 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc04c020 con 0x563bfad801a0 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbad7fa640 1 -- 172.21.15.73:0/2139893409 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc02cb60 con 0x563bfad801a0 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbad7fa640 1 -- 172.21.15.73:0/2139893409 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc0478d0 con 0x563bfad801a0 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 -- 172.21.15.73:0/2139893409 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 msgr2=0x563bfad88db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc036040 tx=0x7fbbbc0092d0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 -- 172.21.15.73:0/2139893409 shutdown_connections 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad896a0 0x563bfad83e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad80ac0 0x563bfad838e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 --2- 172.21.15.73:0/2139893409 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad88db0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 -- 172.21.15.73:0/2139893409 >> 172.21.15.73:0/2139893409 conn(0x563bfad7a8d0 msgr2=0x563bfad7acb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 -- 172.21.15.73:0/2139893409 shutdown_connections 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.794+0000 7fbbd040e740 1 -- 172.21.15.73:0/2139893409 wait complete. 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 -- start start 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 Processor -- start 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 0x563bfad8ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8f120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81ad430 con 0x563bfad801a0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81ae750 con 0x563bfad80ac0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbc81ada90 con 0x563bfad896a0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbd040e740 10 client.0 _ll_register_callbacks cb 0x7fbbb800c350 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40964/0 (socket says 172.21.15.73:40964) 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 -- 172.21.15.73:0/300794762 learned_addr learned my addr 172.21.15.73:0/300794762 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 0x563bfad8ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 -- 172.21.15.73:0/300794762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 msgr2=0x563bfad8f120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8f120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 -- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 msgr2=0x563bfad8ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 0x563bfad8ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc67fc640 1 -- 172.21.15.73:0/300794762 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563bfad8f6e0 con 0x563bfad801a0 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.795+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 0x563bfad8ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbc67fc640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009900 tx=0x7fbbbc0474b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a9adb89ef1ec65 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:26.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbbbc04c030 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fbbbc004040 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbbbc02cb60 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbbbc004240 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 10 client.16299 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563bfadf0790 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbbbc02cb60 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbbbc043590 con 0x563bfad801a0 2024-03-09T00:10:26.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 10 client.16299 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 10 client.16299 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 20 client.16299 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfadf0e60 0x563bfadf3280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 4 client.16299 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.796+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x563bfaceb750 con 0x563bfadf0e60 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfadf0e60 0x563bfadf3280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbba6ffd640 10 client.16299 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbc6ffd640 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfadf0e60 0x563bfadf3280 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6d1246786159c075 server_cookie=a1f49237f7875ddb in_seq=0 out_seq=0 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbba6ffd640 1 -- 172.21.15.73:0/300794762 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x563bfaceb750 con 0x563bfadf0e60 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbba6ffd640 10 client.16299 handle_command_reply: tid=0 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 client.16299 shutdown 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 msgr2=0x563bfad90840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 crypto rx=0x7fbbbc009900 tx=0x7fbbbc0474b0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 shutdown_connections 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563bfadf0e60 0x563bfadf3280 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563bfad896a0 0x563bfad8f120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbba6ffd640 0 client.16299 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563bfad80ac0 0x563bfad8ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 --2- 172.21.15.73:0/300794762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563bfad801a0 0x563bfad90840 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 >> 172.21.15.73:0/300794762 conn(0x563bfad7a8d0 msgr2=0x563bfad84f50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 shutdown_connections 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 1 -- 172.21.15.73:0/300794762 wait complete. 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.797+0000 7fbbd040e740 20 client.16299 trim_cache size 0 max 16384 2024-03-09T00:10:26.808 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:10:26.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 msgr2=0x7fbbb00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 0x7fbbb00a4e30 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fbbc00a3df0 tx=0x7fbbc006a990 comp rx=0 tx=0).stop 2024-03-09T00:10:26.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 msgr2=0x7fbbc81a39d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:26.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 crypto rx=0x7fbbc000eb10 tx=0x7fbbc000c2d0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 shutdown_connections 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbbb00a2970 0x7fbbb00a4e30 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbbc8106310 0x7fbbc81a39d0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbbc8100e00 0x7fbbc8198e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 --2- 172.21.15.73:0/3168675850 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbbc8100440 0x7fbbc8198950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 >> 172.21.15.73:0/3168675850 conn(0x7fbbc8076f80 msgr2=0x7fbbc80ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 shutdown_connections 2024-03-09T00:10:26.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:26.798+0000 7fbbcd832640 1 -- 172.21.15.73:0/3168675850 wait complete. 2024-03-09T00:10:27.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:27 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1886444816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:27.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:27 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1886444816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:27.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:27 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1886444816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:27.813 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:10:27.813 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:10:27.813 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:10:27.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- 172.21.15.73:0/2533543091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2780100410 msgr2=0x7f27801007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 --2- 172.21.15.73:0/2533543091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2780100410 0x7f27801007f0 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 crypto rx=0x7f2770009900 tx=0x7f277002bc30 comp rx=0 tx=0).stop 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- 172.21.15.73:0/2533543091 shutdown_connections 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 --2- 172.21.15.73:0/2533543091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27801054e0 0x7f278010d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 --2- 172.21.15.73:0/2533543091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100dc0 0x7f2780104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 --2- 172.21.15.73:0/2533543091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2780100410 0x7f27801007f0 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- 172.21.15.73:0/2533543091 >> 172.21.15.73:0/2533543091 conn(0x7f2780076570 msgr2=0x7f2780076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- 172.21.15.73:0/2533543091 shutdown_connections 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- 172.21.15.73:0/2533543091 wait complete. 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 Processor -- start 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.936+0000 7f278629a640 1 -- start start 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100410 0x7f2780198930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2780199400 0x7f27801a39b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2780110570 con 0x7f27801054e0 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801103f0 con 0x7f2780199400 2024-03-09T00:10:27.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f278629a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801106f0 con 0x7f2780100410 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40990/0 (socket says 172.21.15.73:40990) 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 -- 172.21.15.73:0/651201700 learned_addr learned my addr 172.21.15.73:0/651201700 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277ffff640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100410 0x7f2780198930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f2784837640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2780199400 0x7f27801a39b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 -- 172.21.15.73:0/651201700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100410 msgr2=0x7f2780198930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100410 0x7f2780198930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 -- 172.21.15.73:0/651201700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2780199400 msgr2=0x7f27801a39b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2780199400 0x7f27801a39b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 -- 172.21.15.73:0/651201700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27801a41b0 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.937+0000 7f277f7fe640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f277400c970 tx=0x7f277400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.938+0000 7f277d7fa640 1 -- 172.21.15.73:0/651201700 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2774014070 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.938+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27801a44a0 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.938+0000 7f277d7fa640 1 -- 172.21.15.73:0/651201700 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2774004440 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.938+0000 7f277d7fa640 1 -- 172.21.15.73:0/651201700 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277400f450 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.938+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27801a4a40 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.940+0000 7f277d7fa640 1 -- 172.21.15.73:0/651201700 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2774017070 con 0x7f27801054e0 2024-03-09T00:10:27.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.940+0000 7f2788e82740 1 Processor -- start 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.940+0000 7f2788e82740 1 -- start start 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.940+0000 7f277d7fa640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 0x7f27680a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c772e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c780150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c6d6b40 con 0x55e32c780690 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c67fd80 con 0x55e32c7733d0 2024-03-09T00:10:27.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2788e82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c6d2be0 con 0x55e32c772ab0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 0x7f27680a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f277d7fa640 1 -- 172.21.15.73:0/651201700 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f27740cf0b0 con 0x7f27801054e0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:40998/0 (socket says 172.21.15.73:40998) 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 -- 172.21.15.73:0/2994743266 learned_addr learned my addr 172.21.15.73:0/2994743266 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c780150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f277ffff640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c772e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 0x7f27680a4eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2778013770 tx=0x7f2778013b70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 -- 172.21.15.73:0/2994743266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 msgr2=0x55e32c772e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c772e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 -- 172.21.15.73:0/2994743266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 msgr2=0x55e32c780150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c780150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 -- 172.21.15.73:0/2994743266 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e32c4b37e0 con 0x55e32c780690 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c780150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f2784837640 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2778009d20 tx=0x7f277801d000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ad262dfe96b9f1a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.941+0000 7f27667fc640 1 -- 172.21.15.73:0/2994743266 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277800d9d0 con 0x55e32c780690 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f27667fc640 1 -- 172.21.15.73:0/2994743266 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277800db70 con 0x55e32c780690 2024-03-09T00:10:27.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f27667fc640 1 -- 172.21.15.73:0/2994743266 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277801e630 con 0x55e32c780690 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 -- 172.21.15.73:0/2994743266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 msgr2=0x55e32c782a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 crypto rx=0x7f2778009d20 tx=0x7f277801d000 comp rx=0 tx=0).stop 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 -- 172.21.15.73:0/2994743266 shutdown_connections 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c782a80 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c780150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 --2- 172.21.15.73:0/2994743266 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c772e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 -- 172.21.15.73:0/2994743266 >> 172.21.15.73:0/2994743266 conn(0x55e32c76d4b0 msgr2=0x55e32c76d890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 -- 172.21.15.73:0/2994743266 shutdown_connections 2024-03-09T00:10:27.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.942+0000 7f2788e82740 1 -- 172.21.15.73:0/2994743266 wait complete. 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 -- start start 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 Processor -- start 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c77d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c780690 0x55e32c77da50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c7833f0 con 0x55e32c772ab0 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c6931c0 con 0x55e32c7733d0 2024-03-09T00:10:27.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e32c783570 con 0x55e32c780690 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2788e82740 10 client.0 _ll_register_callbacks cb 0x55e32c6827a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41000/0 (socket says 172.21.15.73:41000) 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 -- 172.21.15.73:0/2331474802 learned_addr learned my addr 172.21.15.73:0/2331474802 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f2784837640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c780690 0x55e32c77da50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 -- 172.21.15.73:0/2331474802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c780690 msgr2=0x55e32c77da50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c780690 0x55e32c77da50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c77d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 -- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 msgr2=0x55e32c77d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c77d440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.943+0000 7f277ffff640 1 -- 172.21.15.73:0/2331474802 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e32c513f70 con 0x55e32c772ab0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c77d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f277ffff640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f277002c170 tx=0x7f277002ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=baa510b8eb484d1e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f277002c690 con 0x55e32c772ab0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f27700046c0 con 0x55e32c772ab0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f277800db70 con 0x55e32c772ab0 2024-03-09T00:10:27.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f27700048c0 con 0x55e32c772ab0 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 10 client.16311 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e32c3ab170 con 0x55e32c772ab0 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f277800db70 con 0x55e32c772ab0 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2770003710 con 0x55e32c772ab0 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 10 client.16311 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 10 client.16311 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 20 client.16311 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c790780 0x55e32c792b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 4 client.16311 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:27.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.944+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e32c6ac960 con 0x55e32c790780 2024-03-09T00:10:27.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.945+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c790780 0x55e32c792b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.945+0000 7f275f7fe640 10 client.16311 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:27.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.945+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c790780 0x55e32c792b70 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=57f9383775d00761 server_cookie=ed7842722ec750ac in_seq=0 out_seq=0 2024-03-09T00:10:27.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f275f7fe640 1 -- 172.21.15.73:0/2331474802 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e32c6ac960 con 0x55e32c790780 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f275f7fe640 10 client.16311 handle_command_reply: tid=0 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 client.16311 shutdown 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 msgr2=0x55e32c7800b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 crypto rx=0x7f277002c170 tx=0x7f277002ce30 comp rx=0 tx=0).stop 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 shutdown_connections 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c790780 0x55e32c792b70 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c780690 0x55e32c77da50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c7733d0 0x55e32c77d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f275f7fe640 0 client.16311 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 --2- 172.21.15.73:0/2331474802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c772ab0 0x55e32c7800b0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 >> 172.21.15.73:0/2331474802 conn(0x55e32c76d4b0 msgr2=0x55e32c76f200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 shutdown_connections 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 1 -- 172.21.15.73:0/2331474802 wait complete. 2024-03-09T00:10:27.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.946+0000 7f2788e82740 20 client.16311 trim_cache size 0 max 16384 2024-03-09T00:10:27.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 Processor -- start 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 -- start start 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c77c420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 0x55e32c7800b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801aa4c0 con 0x55e32c7733d0 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801aca80 con 0x55e32c780690 2024-03-09T00:10:27.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.961+0000 7f2788e82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801aaac0 con 0x55e32c772ab0 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41008/0 (socket says 172.21.15.73:41008) 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 -- 172.21.15.73:0/1930336347 learned_addr learned my addr 172.21.15.73:0/1930336347 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277ffff640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c77c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f2784837640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 0x55e32c7800b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 -- 172.21.15.73:0/1930336347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 msgr2=0x55e32c77c420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c77c420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 -- 172.21.15.73:0/1930336347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 msgr2=0x55e32c7800b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.965 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 0x55e32c7800b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 -- 172.21.15.73:0/1930336347 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e32c77ddc0 con 0x55e32c7733d0 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277ffff640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c77c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f2784837640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 0x55e32c7800b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f277f7fe640 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2774097730 tx=0x7f2774007f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=23311e444eb0330d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f2765ffb640 1 -- 172.21.15.73:0/1930336347 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27740a3ac0 con 0x55e32c7733d0 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f2765ffb640 1 -- 172.21.15.73:0/1930336347 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2774099620 con 0x55e32c7733d0 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.962+0000 7f2765ffb640 1 -- 172.21.15.73:0/1930336347 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27740a3630 con 0x55e32c7733d0 2024-03-09T00:10:27.966 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- 172.21.15.73:0/1930336347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 msgr2=0x55e32c77fb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 crypto rx=0x7f2774097730 tx=0x7f2774007f60 comp rx=0 tx=0).stop 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- 172.21.15.73:0/1930336347 shutdown_connections 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c780690 0x55e32c7800b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c7733d0 0x55e32c77fb70 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- 172.21.15.73:0/1930336347 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c772ab0 0x55e32c77c420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- 172.21.15.73:0/1930336347 >> 172.21.15.73:0/1930336347 conn(0x55e32c76d4b0 msgr2=0x55e32c77bd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- 172.21.15.73:0/1930336347 shutdown_connections 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- 172.21.15.73:0/1930336347 wait complete. 2024-03-09T00:10:27.967 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 -- start start 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 Processor -- start 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c772ab0 0x55e32c7e0fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c7733d0 0x55e32c7e14f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.963+0000 7f2788e82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801ab100 con 0x55e32c780690 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801ac420 con 0x55e32c772ab0 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27801ab760 con 0x55e32c7733d0 2024-03-09T00:10:27.968 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 10 client.0 _ll_register_callbacks cb 0x7f2778009c50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41014/0 (socket says 172.21.15.73:41014) 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 -- 172.21.15.73:0/2041963806 learned_addr learned my addr 172.21.15.73:0/2041963806 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f277ffff640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c772ab0 0x55e32c7e0fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f277f7fe640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c7733d0 0x55e32c7e14f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 -- 172.21.15.73:0/2041963806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c7733d0 msgr2=0x55e32c7e14f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c7733d0 0x55e32c7e14f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 -- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c772ab0 msgr2=0x55e32c7e0fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:27.969 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c772ab0 0x55e32c7e0fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 -- 172.21.15.73:0/2041963806 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e32c792ee0 con 0x55e32c780690 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2784837640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x55e32c76e640 tx=0x7f2778011f50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f1ce4cd376f639ce server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f275f7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2778013020 con 0x55e32c780690 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f27780131c0 con 0x55e32c780690 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2774099620 con 0x55e32c780690 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2778004bb0 con 0x55e32c780690 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 10 client.16317 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:27.970 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.964+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e32c7e8540 con 0x55e32c780690 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2774099620 con 0x55e32c780690 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2778005270 con 0x55e32c780690 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 10 client.16317 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 10 client.16317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 20 client.16317 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c7e8a60 0x55e32c7eae80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 4 client.16317 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:10:27.971 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e32c698080 con 0x55e32c7e8a60 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f277ffff640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c7e8a60 0x55e32c7eae80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f275f7fe640 10 client.16317 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.965+0000 7f277ffff640 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c7e8a60 0x55e32c7eae80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5a822010400719b5 server_cookie=43f8ccf976d038f1 in_seq=0 out_seq=0 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.966+0000 7f275f7fe640 1 -- 172.21.15.73:0/2041963806 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e32c698080 con 0x55e32c7e8a60 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.966+0000 7f275f7fe640 10 client.16317 handle_command_reply: tid=0 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.966+0000 7f2788e82740 1 client.16317 shutdown 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.966+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 msgr2=0x55e32c784a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.966+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 crypto rx=0x55e32c76e640 tx=0x7f2778011f50 comp rx=0 tx=0).stop 2024-03-09T00:10:27.972 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 shutdown_connections 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e32c7e8a60 0x55e32c7eae80 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e32c780690 0x55e32c784a90 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e32c7733d0 0x55e32c7e14f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f275f7fe640 0 client.16317 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 --2- 172.21.15.73:0/2041963806 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e32c772ab0 0x55e32c7e0fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 >> 172.21.15.73:0/2041963806 conn(0x55e32c76d4b0 msgr2=0x55e32c77c5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 shutdown_connections 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 1 -- 172.21.15.73:0/2041963806 wait complete. 2024-03-09T00:10:27.973 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f2788e82740 20 client.16317 trim_cache size 0 max 16384 2024-03-09T00:10:27.974 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:10:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:10:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "6fbb0c59-fc49-456e-b670-18c0001745ea", 2024-03-09T00:10:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:10:27.974 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 msgr2=0x7f27680a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 0x7f27680a4eb0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f2778013770 tx=0x7f2778013b70 comp rx=0 tx=0).stop 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 msgr2=0x7f2780198e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 crypto rx=0x7f277400c970 tx=0x7f277400ce30 comp rx=0 tx=0).stop 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 shutdown_connections 2024-03-09T00:10:27.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2780199400 0x7f27801a39b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27680a29f0 0x7f27680a4eb0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27801054e0 0x7f2780198e70 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 --2- 172.21.15.73:0/651201700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2780100410 0x7f2780198930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 >> 172.21.15.73:0/651201700 conn(0x7f2780076570 msgr2=0x7f27800fdfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 shutdown_connections 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:27.967+0000 7f278629a640 1 -- 172.21.15.73:0/651201700 wait complete. 2024-03-09T00:10:27.977 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:10:27.978 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:10:28.102 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 -- 172.21.15.73:0/4064485510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c44100d80 msgr2=0x7f0c44104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 --2- 172.21.15.73:0/4064485510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c44100d80 0x7f0c44104dc0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009860 tx=0x7f0c3802bb70 comp rx=0 tx=0).stop 2024-03-09T00:10:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 -- 172.21.15.73:0/4064485510 shutdown_connections 2024-03-09T00:10:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 --2- 172.21.15.73:0/4064485510 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441054a0 0x7f0c4410d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 --2- 172.21.15.73:0/4064485510 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c44100d80 0x7f0c44104dc0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.103 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 --2- 172.21.15.73:0/4064485510 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.092+0000 7f0c4a553640 1 -- 172.21.15.73:0/4064485510 >> 172.21.15.73:0/4064485510 conn(0x7f0c440772f0 msgr2=0x7f0c44077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.093+0000 7f0c4a553640 1 -- 172.21.15.73:0/4064485510 shutdown_connections 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.093+0000 7f0c4a553640 1 -- 172.21.15.73:0/4064485510 wait complete. 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.093+0000 7f0c4a553640 1 Processor -- start 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.093+0000 7f0c4a553640 1 -- start start 2024-03-09T00:10:28.104 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c441054a0 0x7f0c44198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 0x7f0c441a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c44110530 con 0x7f0c441994f0 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441103b0 con 0x7f0c441054a0 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c4a553640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441106b0 con 0x7f0c441003d0 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 0x7f0c441a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c437fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c441054a0 0x7f0c44198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:39242/0 (socket says 172.21.15.73:39242) 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c43fff640 1 -- 172.21.15.73:0/158437009 learned_addr learned my addr 172.21.15.73:0/158437009 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 -- 172.21.15.73:0/158437009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 msgr2=0x7f0c441989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 -- 172.21.15.73:0/158437009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c441054a0 msgr2=0x7f0c44198f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c441054a0 0x7f0c44198f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 -- 172.21.15.73:0/158437009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c441a4270 con 0x7f0c441994f0 2024-03-09T00:10:28.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c43fff640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c48af0640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 0x7f0c441a3a50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00ef00 tx=0x7f0c3c00c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.094+0000 7f0c417fa640 1 -- 172.21.15.73:0/158437009 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c004b20 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.095+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c441a4560 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.095+0000 7f0c417fa640 1 -- 172.21.15.73:0/158437009 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3c003e50 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.095+0000 7f0c417fa640 1 -- 172.21.15.73:0/158437009 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3c00b570 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.095+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c441969a0 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.097+0000 7f0c417fa640 1 -- 172.21.15.73:0/158437009 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0c3c01d020 con 0x7f0c441994f0 2024-03-09T00:10:28.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 Processor -- start 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 -- start start 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c417fa640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 0x7f0c2c0a9240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c417fa640 1 -- 172.21.15.73:0/158437009 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0c3c0ceff0 con 0x7f0c441994f0 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c437fe640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 0x7f0c2c0a9240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d564ae00 con 0x5640d56e7760 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d55f3b10 con 0x5640d56f0c60 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c4d211740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d5646ea0 con 0x5640d56e8080 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41048/0 (socket says 172.21.15.73:41048) 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 -- 172.21.15.73:0/277213930 learned_addr learned my addr 172.21.15.73:0/277213930 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c48af0640 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c437fe640 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 -- 172.21.15.73:0/277213930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 msgr2=0x5640d56f0720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f0720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 -- 172.21.15.73:0/277213930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 msgr2=0x5640d56f3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.098+0000 7f0c43fff640 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c43fff640 1 -- 172.21.15.73:0/277213930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640d54297d0 con 0x5640d56e7760 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c437fe640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 0x7f0c2c0a9240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009830 tx=0x7f0c38034cf0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c43fff640 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0c34011630 tx=0x7f0c34011a30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=27614898aa45b146 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c2a7fc640 1 -- 172.21.15.73:0/277213930 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3401e070 con 0x5640d56e7760 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c2a7fc640 1 -- 172.21.15.73:0/277213930 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3400fb20 con 0x5640d56e7760 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.099+0000 7f0c2a7fc640 1 -- 172.21.15.73:0/277213930 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c34019cd0 con 0x5640d56e7760 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- 172.21.15.73:0/277213930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 msgr2=0x5640d56e7b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 crypto rx=0x7f0c34011630 tx=0x7f0c34011a30 comp rx=0 tx=0).stop 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- 172.21.15.73:0/277213930 shutdown_connections 2024-03-09T00:10:28.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f0720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 --2- 172.21.15.73:0/277213930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56e7b40 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- 172.21.15.73:0/277213930 >> 172.21.15.73:0/277213930 conn(0x5640d56e1e90 msgr2=0x5640d56e2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- 172.21.15.73:0/277213930 shutdown_connections 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- 172.21.15.73:0/277213930 wait complete. 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 -- start start 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.100+0000 7f0c4d211740 1 Processor -- start 2024-03-09T00:10:28.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56e8080 0x5640d56ed6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56f0c60 0x5640d56edc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d56f3a50 con 0x5640d56e7760 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d56070a0 con 0x5640d56e8080 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5640d56f3bd0 con 0x5640d56f0c60 2024-03-09T00:10:28.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c4d211740 10 client.0 _ll_register_callbacks cb 0x5640d5606a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41062/0 (socket says 172.21.15.73:41062) 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 -- 172.21.15.73:0/2826360952 learned_addr learned my addr 172.21.15.73:0/2826360952 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c437fe640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56e8080 0x5640d56ed6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c48af0640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56f0c60 0x5640d56edc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 -- 172.21.15.73:0/2826360952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56f0c60 msgr2=0x5640d56edc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56f0c60 0x5640d56edc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 -- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56e8080 msgr2=0x5640d56ed6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56e8080 0x5640d56ed6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 -- 172.21.15.73:0/2826360952 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640d535f3f0 con 0x5640d56e7760 2024-03-09T00:10:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c43fff640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0c340198f0 tx=0x7f0c34009f10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=220bd2ab64390ca6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.101+0000 7f0c237fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3401e040 con 0x5640d56e7760 2024-03-09T00:10:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0c3400f740 con 0x5640d56e7760 2024-03-09T00:10:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c3400fb20 con 0x5640d56e7760 2024-03-09T00:10:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c34030050 con 0x5640d56e7760 2024-03-09T00:10:28.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 10 client.16329 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5640d531f0f0 con 0x5640d56e7760 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c3400fb20 con 0x5640d56e7760 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c34021070 con 0x5640d56e7760 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 10 client.16329 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 10 client.16329 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 20 client.16329 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:28.117 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d5702a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 4 client.16329 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.102+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5640d5658340 con 0x5640d5700640 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.103+0000 7f0c437fe640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d5702a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.103+0000 7f0c437fe640 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d5702a30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61e4be3dd97a22ef server_cookie=4d39f3cb7a82f123 in_seq=0 out_seq=0 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.103+0000 7f0c237fe640 10 client.16329 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c237fe640 1 -- 172.21.15.73:0/2826360952 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5640d5658340 con 0x5640d5700640 2024-03-09T00:10:28.118 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c237fe640 10 client.16329 handle_command_reply: tid=0 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 client.16329 shutdown 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 msgr2=0x5640d56f0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 crypto rx=0x7f0c340198f0 tx=0x7f0c34009f10 comp rx=0 tx=0).stop 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 shutdown_connections 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d5702a30 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56f0c60 0x5640d56edc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c237fe640 0 client.16329 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:28.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56e8080 0x5640d56ed6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 --2- 172.21.15.73:0/2826360952 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f0360 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 >> 172.21.15.73:0/2826360952 conn(0x5640d56e1e90 msgr2=0x5640d56e3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 shutdown_connections 2024-03-09T00:10:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 1 -- 172.21.15.73:0/2826360952 wait complete. 2024-03-09T00:10:28.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.104+0000 7f0c4d211740 20 client.16329 trim_cache size 0 max 16384 2024-03-09T00:10:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 Processor -- start 2024-03-09T00:10:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 -- start start 2024-03-09T00:10:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e7760 0x5640d56efe20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.127 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f7e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441aa4a0 con 0x5640d56e8080 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.121+0000 7f0c4d211740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441aca60 con 0x5640d56f0c60 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c4d211740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441aaaa0 con 0x5640d56e7760 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41070/0 (socket says 172.21.15.73:41070) 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 -- 172.21.15.73:0/3430171244 learned_addr learned my addr 172.21.15.73:0/3430171244 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:28.128 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c48af0640 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f7e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c43fff640 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e7760 0x5640d56efe20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 -- 172.21.15.73:0/3430171244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e7760 msgr2=0x5640d56efe20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e7760 0x5640d56efe20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 -- 172.21.15.73:0/3430171244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 msgr2=0x5640d56f7e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f7e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 -- 172.21.15.73:0/3430171244 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5640d56f3930 con 0x5640d56e8080 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c437fe640 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0c380095f0 tx=0x7f0c3802e050 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4fa669a72e2c5792 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.129 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c29ffb640 1 -- 172.21.15.73:0/3430171244 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38055070 con 0x5640d56e8080 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c29ffb640 1 -- 172.21.15.73:0/3430171244 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c38047040 con 0x5640d56e8080 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.122+0000 7f0c29ffb640 1 -- 172.21.15.73:0/3430171244 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c38039040 con 0x5640d56e8080 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- 172.21.15.73:0/3430171244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 msgr2=0x5640d56f0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 crypto rx=0x7f0c380095f0 tx=0x7f0c3802e050 comp rx=0 tx=0).stop 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- 172.21.15.73:0/3430171244 shutdown_connections 2024-03-09T00:10:28.130 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56f7e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e8080 0x5640d56f0360 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 --2- 172.21.15.73:0/3430171244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e7760 0x5640d56efe20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- 172.21.15.73:0/3430171244 >> 172.21.15.73:0/3430171244 conn(0x5640d56e1e90 msgr2=0x5640d56e2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- 172.21.15.73:0/3430171244 shutdown_connections 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- 172.21.15.73:0/3430171244 wait complete. 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.123+0000 7f0c4d211740 1 -- start start 2024-03-09T00:10:28.131 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 Processor -- start 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f8350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56e9eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441ab0e0 con 0x5640d56e7760 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441ac400 con 0x5640d56f0c60 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c441ab740 con 0x5640d56e8080 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c4d211740 10 client.0 _ll_register_callbacks cb 0x7f0c3401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:28.132 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41072/0 (socket says 172.21.15.73:41072) 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 -- 172.21.15.73:0/443360178 learned_addr learned my addr 172.21.15.73:0/443360178 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c437fe640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c48af0640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56e9eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 -- 172.21.15.73:0/443360178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 msgr2=0x5640d56f8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 -- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 msgr2=0x5640d56e9eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56e9eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.124+0000 7f0c43fff640 1 -- 172.21.15.73:0/443360178 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5640d56ea6e0 con 0x5640d56e7760 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c43fff640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009640 tx=0x7f0c3400f3c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=721f77319718861d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0c3401e040 con 0x5640d56e7760 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f0c3400f840 con 0x5640d56e7760 2024-03-09T00:10:28.134 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0c38047040 con 0x5640d56e7760 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0c3400fa40 con 0x5640d56e7760 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 10 client.16335 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5640d575fe30 con 0x5640d56e7760 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0c38047040 con 0x5640d56e7760 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0c34021070 con 0x5640d56e7760 2024-03-09T00:10:28.135 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 10 client.16335 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 10 client.16335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 20 client.16335 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d57625c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 4 client.16335 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5640d5652690 con 0x5640d5700640 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.125+0000 7f0c437fe640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d57625c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c437fe640 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d57625c0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f763210e3c95c996 server_cookie=382e832e0a933239 in_seq=0 out_seq=0 2024-03-09T00:10:28.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c237fe640 10 client.16335 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c237fe640 1 -- 172.21.15.73:0/443360178 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5640d5652690 con 0x5640d5700640 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c237fe640 10 client.16335 handle_command_reply: tid=0 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 client.16335 shutdown 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 msgr2=0x5640d56f7e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 crypto rx=0x7f0c34009640 tx=0x7f0c3400f3c0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.137 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 shutdown_connections 2024-03-09T00:10:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5640d5700640 0x5640d57625c0 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5640d56f0c60 0x5640d56e9eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5640d56e8080 0x5640d56f8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c237fe640 0 client.16335 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:28.138 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 --2- 172.21.15.73:0/443360178 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5640d56e7760 0x5640d56f7e10 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 >> 172.21.15.73:0/443360178 conn(0x5640d56e1e90 msgr2=0x5640d56ebfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 shutdown_connections 2024-03-09T00:10:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 1 -- 172.21.15.73:0/443360178 wait complete. 2024-03-09T00:10:28.139 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.126+0000 7f0c4d211740 20 client.16335 trim_cache size 0 max 16384 2024-03-09T00:10:28.139 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2326 inodes in the stack)", 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "6fbb0c59-fc49-456e-b670-18c0001745ea": { 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "6fbb0c59-fc49-456e-b670-18c0001745ea", 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:10:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:10:28.141 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:10:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 msgr2=0x7f0c2c0a9240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 0x7f0c2c0a9240 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f0c38009830 tx=0x7f0c38034cf0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 msgr2=0x7f0c441a3a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:28.143 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 0x7f0c441a3a50 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 crypto rx=0x7f0c3c00ef00 tx=0x7f0c3c00c740 comp rx=0 tx=0).stop 2024-03-09T00:10:28.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 shutdown_connections 2024-03-09T00:10:28.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0c2c0a6d80 0x7f0c2c0a9240 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0c441994f0 0x7f0c441a3a50 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0c441054a0 0x7f0c44198f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 --2- 172.21.15.73:0/158437009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0c441003d0 0x7f0c441989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 >> 172.21.15.73:0/158437009 conn(0x7f0c440772f0 msgr2=0x7f0c440fe390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 shutdown_connections 2024-03-09T00:10:28.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:28.127+0000 7f0c4a553640 1 -- 172.21.15.73:0/158437009 wait complete. 2024-03-09T00:10:28.147 INFO:tasks.cephfs.filesystem:scrub status for tag:6fbb0c59-fc49-456e-b670-18c0001745ea - {'path': '/', 'tag': '6fbb0c59-fc49-456e-b670-18c0001745ea', 'options': 'recursive,force'} 2024-03-09T00:10:28.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:28 smithi073 ceph-mon[32309]: pgmap v111: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:28 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:10:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:28 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:28 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:10:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:28 smithi102 ceph-mon[36106]: pgmap v111: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:28 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:10:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:28 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:28.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:28 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:10:28.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:28 smithi179 ceph-mon[36637]: pgmap v111: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 1.6 KiB/s wr, 2 op/s 2024-03-09T00:10:28.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:28 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:10:28.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:28 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:28.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:28 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:10:29.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:29 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:29.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:29 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:10:29.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:29 smithi073 ceph-mon[32309]: pgmap v112: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:29.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:29 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:29.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:29 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:10:29.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:29 smithi102 ceph-mon[36106]: pgmap v112: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:29.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:29 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:10:29.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:29 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:10:29.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:29 smithi179 ceph-mon[36637]: pgmap v112: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:32.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:32 smithi179 ceph-mon[36637]: pgmap v113: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:32.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:32 smithi073 ceph-mon[32309]: pgmap v113: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:32.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:32 smithi102 ceph-mon[36106]: pgmap v113: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:34.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:34 smithi179 ceph-mon[36637]: pgmap v114: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:34 smithi073 ceph-mon[32309]: pgmap v114: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:34.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:34 smithi102 ceph-mon[36106]: pgmap v114: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:36.332 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:36 smithi179 ceph-mon[36637]: pgmap v115: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:36.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:36 smithi073 ceph-mon[32309]: pgmap v115: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:36.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:36 smithi102 ceph-mon[36106]: pgmap v115: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:38.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:38 smithi073 ceph-mon[32309]: pgmap v116: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:38.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:38 smithi102 ceph-mon[36106]: pgmap v116: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:38.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:38 smithi179 ceph-mon[36637]: pgmap v116: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:40.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:40 smithi073 ceph-mon[32309]: pgmap v117: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:40.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:40 smithi102 ceph-mon[36106]: pgmap v117: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:40.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:40 smithi179 ceph-mon[36637]: pgmap v117: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:42.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:42 smithi073 ceph-mon[32309]: pgmap v118: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:42.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:42 smithi102 ceph-mon[36106]: pgmap v118: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:42.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:42 smithi179 ceph-mon[36637]: pgmap v118: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:10:44.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:44 smithi073 ceph-mon[32309]: pgmap v119: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:44.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:44 smithi102 ceph-mon[36106]: pgmap v119: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:44.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:44 smithi179 ceph-mon[36637]: pgmap v119: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:10:46.384 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:46 smithi179 ceph-mon[36637]: pgmap v120: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:46.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:46 smithi073 ceph-mon[32309]: pgmap v120: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:46.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:46 smithi102 ceph-mon[36106]: pgmap v120: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:48.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:48 smithi073 ceph-mon[32309]: pgmap v121: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:48.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:48 smithi102 ceph-mon[36106]: pgmap v121: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:48.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:48 smithi179 ceph-mon[36637]: pgmap v121: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:50.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:50 smithi073 ceph-mon[32309]: pgmap v122: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:50.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:50 smithi102 ceph-mon[36106]: pgmap v122: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:50.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:50 smithi179 ceph-mon[36637]: pgmap v122: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:52 smithi073 ceph-mon[32309]: pgmap v123: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:52.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:52 smithi102 ceph-mon[36106]: pgmap v123: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:52.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:52 smithi179 ceph-mon[36637]: pgmap v123: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:54.401 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:54 smithi073 ceph-mon[32309]: pgmap v124: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:54.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:54 smithi102 ceph-mon[36106]: pgmap v124: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:54.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:54 smithi179 ceph-mon[36637]: pgmap v124: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:56 smithi073 ceph-mon[32309]: pgmap v125: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:56 smithi102 ceph-mon[36106]: pgmap v125: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:56.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:56 smithi179 ceph-mon[36637]: pgmap v125: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:10:58.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:10:58.149 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:10:58.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- 172.21.15.73:0/133504300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41901065a0 msgr2=0x7f419010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 --2- 172.21.15.73:0/133504300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41901065a0 0x7f419010e170 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 crypto rx=0x7f4188009ca0 tx=0x7f418802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- 172.21.15.73:0/133504300 shutdown_connections 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 --2- 172.21.15.73:0/133504300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f41901065a0 0x7f419010e170 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 --2- 172.21.15.73:0/133504300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41900690f0 0x7f4190105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 --2- 172.21.15.73:0/133504300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 0x7f4190068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- 172.21.15.73:0/133504300 >> 172.21.15.73:0/133504300 conn(0x7f41900772f0 msgr2=0x7f4190077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- 172.21.15.73:0/133504300 shutdown_connections 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- 172.21.15.73:0/133504300 wait complete. 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 Processor -- start 2024-03-09T00:10:58.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.270+0000 7f4198644640 1 -- start start 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 0x7f41901001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41901065a0 0x7f41901006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4190111660 con 0x7f4190100c30 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901114e0 con 0x7f4190068740 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901117e0 con 0x7f41901065a0 2024-03-09T00:10:58.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33720/0 (socket says 172.21.15.73:33720) 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 -- 172.21.15.73:0/224866738 learned_addr learned my addr 172.21.15.73:0/224866738 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f41963e0640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 0x7f41901001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4195bdf640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41901065a0 0x7f41901006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 -- 172.21.15.73:0/224866738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41901065a0 msgr2=0x7f41901006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41901065a0 0x7f41901006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 -- 172.21.15.73:0/224866738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 msgr2=0x7f41901001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 0x7f41901001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 -- 172.21.15.73:0/224866738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41901a6430 con 0x7f4190100c30 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4196be1640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f418802c700 tx=0x7f4188032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f41877fe640 1 -- 172.21.15.73:0/224866738 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4188039040 con 0x7f4190100c30 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f41877fe640 1 -- 172.21.15.73:0/224866738 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4188044070 con 0x7f4190100c30 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f41877fe640 1 -- 172.21.15.73:0/224866738 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880042a0 con 0x7f4190100c30 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.271+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41901a66c0 con 0x7f4190100c30 2024-03-09T00:10:58.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.272+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41901a6bb0 con 0x7f4190100c30 2024-03-09T00:10:58.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.274+0000 7f41877fe640 1 -- 172.21.15.73:0/224866738 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4188004440 con 0x7f4190100c30 2024-03-09T00:10:58.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.274+0000 7f419b224740 1 Processor -- start 2024-03-09T00:10:58.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 -- start start 2024-03-09T00:10:58.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f41877fe640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 0x7f417c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c230bb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c2314720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c226ee00 con 0x55e6c2314c60 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c2217b10 con 0x55e6c230b760 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f419b224740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c226aea0 con 0x55e6c230c080 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4195bdf640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 0x7f417c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33736/0 (socket says 172.21.15.73:33736) 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 -- 172.21.15.73:0/2703221153 learned_addr learned my addr 172.21.15.73:0/2703221153 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f41877fe640 1 -- 172.21.15.73:0/224866738 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41880f14b0 con 0x7f4190100c30 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f41963e0640 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c230bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 -- 172.21.15.73:0/2703221153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 msgr2=0x55e6c2314720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c2314720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 -- 172.21.15.73:0/2703221153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 msgr2=0x55e6c230bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c230bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 -- 172.21.15.73:0/2703221153 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6c204d7d0 con 0x55e6c2314c60 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4195bdf640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 0x7f417c0a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4180005fd0 tx=0x7f418000b5f0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.275+0000 7f4196be1640 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f418804d000 tx=0x7f418802c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f275b52651d6805 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f4184bfe640 1 -- 172.21.15.73:0/2703221153 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880bd3d0 con 0x55e6c2314c60 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f4184bfe640 1 -- 172.21.15.73:0/2703221153 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4188031ea0 con 0x55e6c2314c60 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f4184bfe640 1 -- 172.21.15.73:0/2703221153 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880bd1f0 con 0x55e6c2314c60 2024-03-09T00:10:58.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 -- 172.21.15.73:0/2703221153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 msgr2=0x55e6c2317050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 crypto rx=0x7f418804d000 tx=0x7f418802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 -- 172.21.15.73:0/2703221153 shutdown_connections 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2317050 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c2314720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 --2- 172.21.15.73:0/2703221153 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c230bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 -- 172.21.15.73:0/2703221153 >> 172.21.15.73:0/2703221153 conn(0x55e6c2305e90 msgr2=0x55e6c2306270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 -- 172.21.15.73:0/2703221153 shutdown_connections 2024-03-09T00:10:58.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.276+0000 7f419b224740 1 -- 172.21.15.73:0/2703221153 wait complete. 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 -- start start 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 Processor -- start 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c2314420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c23117b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c2317810 con 0x55e6c2314c60 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c222b0a0 con 0x55e6c230b760 2024-03-09T00:10:58.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e6c2317990 con 0x55e6c230c080 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f419b224740 10 client.0 _ll_register_callbacks cb 0x55e6c222aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33748/0 (socket says 172.21.15.73:33748) 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 -- 172.21.15.73:0/1596358339 learned_addr learned my addr 172.21.15.73:0/1596358339 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4195bdf640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c23117b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f41963e0640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c2314420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 -- 172.21.15.73:0/1596358339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 msgr2=0x55e6c23117b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c23117b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 -- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 msgr2=0x55e6c2314420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c2314420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.277+0000 7f4196be1640 1 -- 172.21.15.73:0/1596358339 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6c1f833f0 con 0x55e6c2314c60 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4196be1640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41880c61f0 tx=0x7f4188035dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=265b278fbb98889f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41880bd3d0 con 0x55e6c2314c60 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f41880a8110 con 0x55e6c2314c60 2024-03-09T00:10:58.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4188031ea0 con 0x55e6c2314c60 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41880bd890 con 0x55e6c2314c60 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 10 client.16347 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e6c1f430f0 con 0x55e6c2314c60 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4188031ea0 con 0x55e6c2314c60 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f418803c030 con 0x55e6c2314c60 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 10 client.16347 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 10 client.16347 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:58.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 20 client.16347 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c2324320 0x55e6c2326710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 4 client.16347 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.278+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e6c227c340 con 0x55e6c2324320 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.279+0000 7f41963e0640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c2324320 0x55e6c2326710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.279+0000 7f4179ffb640 10 client.16347 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.279+0000 7f41963e0640 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c2324320 0x55e6c2326710 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5c8dc54b4d2abc3 server_cookie=709d479489196a2f in_seq=0 out_seq=0 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f4179ffb640 1 -- 172.21.15.73:0/1596358339 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55e6c227c340 con 0x55e6c2324320 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f4179ffb640 10 client.16347 handle_command_reply: tid=0 2024-03-09T00:10:58.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 client.16347 shutdown 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 msgr2=0x55e6c2311cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 crypto rx=0x7f41880c61f0 tx=0x7f4188035dc0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 shutdown_connections 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c2324320 0x55e6c2326710 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c2314c60 0x55e6c2311cf0 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f4179ffb640 0 client.16347 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c23117b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 --2- 172.21.15.73:0/1596358339 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230b760 0x55e6c2314420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 >> 172.21.15.73:0/1596358339 conn(0x55e6c2305e90 msgr2=0x55e6c2307be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 shutdown_connections 2024-03-09T00:10:58.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 1 -- 172.21.15.73:0/1596358339 wait complete. 2024-03-09T00:10:58.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.280+0000 7f419b224740 20 client.16347 trim_cache size 0 max 16384 2024-03-09T00:10:58.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 Processor -- start 2024-03-09T00:10:58.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 -- start start 2024-03-09T00:10:58.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c231bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 0x55e6c2313e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901ae8b0 con 0x55e6c230b760 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901b0e70 con 0x55e6c2314c60 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f419b224740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901aeeb0 con 0x55e6c230c080 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33758/0 (socket says 172.21.15.73:33758) 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 -- 172.21.15.73:0/2302349221 learned_addr learned my addr 172.21.15.73:0/2302349221 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f4196be1640 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 0x55e6c2313e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 -- 172.21.15.73:0/2302349221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 msgr2=0x55e6c231bfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c231bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 -- 172.21.15.73:0/2302349221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 msgr2=0x55e6c2313e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 0x55e6c2313e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f41963e0640 1 -- 172.21.15.73:0/2302349221 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e6c232b720 con 0x55e6c230b760 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.295+0000 7f4196be1640 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 0x55e6c2313e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f41963e0640 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f418c00b890 tx=0x7f418c00b390 comp rx=0 tx=0).ready entity=mon.0 client_cookie=44fd08acec96565f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f417bfff640 1 -- 172.21.15.73:0/2302349221 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c003b60 con 0x55e6c230b760 2024-03-09T00:10:58.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f417bfff640 1 -- 172.21.15.73:0/2302349221 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f418c023020 con 0x55e6c230b760 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f417bfff640 1 -- 172.21.15.73:0/2302349221 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c011860 con 0x55e6c230b760 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 -- 172.21.15.73:0/2302349221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 msgr2=0x55e6c231ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 crypto rx=0x7f418c00b890 tx=0x7f418c00b390 comp rx=0 tx=0).stop 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 -- 172.21.15.73:0/2302349221 shutdown_connections 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c2314c60 0x55e6c2313e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c230c080 0x55e6c231bfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 --2- 172.21.15.73:0/2302349221 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231ba80 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 -- 172.21.15.73:0/2302349221 >> 172.21.15.73:0/2302349221 conn(0x55e6c2305e90 msgr2=0x55e6c2311770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 -- 172.21.15.73:0/2302349221 shutdown_connections 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.296+0000 7f419b224740 1 -- 172.21.15.73:0/2302349221 wait complete. 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 -- start start 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 Processor -- start 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 0x55e6c231a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 0x55e6c231a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901af4f0 con 0x55e6c230b760 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901b0810 con 0x55e6c230c080 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41901afb50 con 0x55e6c2314c60 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f419b224740 10 client.0 _ll_register_callbacks cb 0x7f418c00b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33774/0 (socket says 172.21.15.73:33774) 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 -- 172.21.15.73:0/870926660 learned_addr learned my addr 172.21.15.73:0/870926660 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f4196be1640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 0x55e6c231a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 -- 172.21.15.73:0/870926660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 msgr2=0x55e6c231a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f4195bdf640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 0x55e6c231a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 0x55e6c231a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 -- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 msgr2=0x55e6c231a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 0x55e6c231a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 -- 172.21.15.73:0/870926660 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e6c231ae50 con 0x55e6c230b760 2024-03-09T00:10:58.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f4196be1640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 0x55e6c231a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f4195bdf640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 0x55e6c231a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f41963e0640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f418c00b6d0 tx=0x7f418c003c70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=55bd38899ca212ca server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.297+0000 7f4179ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f418c02c070 con 0x55e6c230b760 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f418c005020 con 0x55e6c230b760 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f418c023020 con 0x55e6c230b760 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f418c037050 con 0x55e6c230b760 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 10 client.16353 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:58.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55e6c237b5a0 con 0x55e6c230b760 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f418c023020 con 0x55e6c230b760 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f418c0047c0 con 0x55e6c230b760 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 10 client.16353 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 10 client.16353 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 20 client.16353 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c237bb90 0x55e6c237dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 4 client.16353 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:10:58.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55e6c2276690 con 0x55e6c237bb90 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.298+0000 7f4195bdf640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c237bb90 0x55e6c237dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f4195bdf640 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c237bb90 0x55e6c237dfb0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a1dd25e9f4562537 server_cookie=a9f1975a404b8986 in_seq=0 out_seq=0 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f4179ffb640 10 client.16353 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f4179ffb640 1 -- 172.21.15.73:0/870926660 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55e6c2276690 con 0x55e6c237bb90 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f4179ffb640 10 client.16353 handle_command_reply: tid=0 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 client.16353 shutdown 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 msgr2=0x55e6c231bda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 crypto rx=0x7f418c00b6d0 tx=0x7f418c003c70 comp rx=0 tx=0).stop 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 shutdown_connections 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55e6c237bb90 0x55e6c237dfb0 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55e6c2314c60 0x55e6c231a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f4179ffb640 0 client.16353 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55e6c230c080 0x55e6c231a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 --2- 172.21.15.73:0/870926660 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55e6c230b760 0x55e6c231bda0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 >> 172.21.15.73:0/870926660 conn(0x55e6c2305e90 msgr2=0x55e6c230ff70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 shutdown_connections 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 1 -- 172.21.15.73:0/870926660 wait complete. 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.299+0000 7f419b224740 20 client.16353 trim_cache size 0 max 16384 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:10:58.308 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:10:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 msgr2=0x7f417c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 0x7f417c0a4e30 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4180005fd0 tx=0x7f418000b5f0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 msgr2=0x7f41901a5d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 crypto rx=0x7f418802c700 tx=0x7f4188032f20 comp rx=0 tx=0).stop 2024-03-09T00:10:58.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 shutdown_connections 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f417c0a2970 0x7f417c0a4e30 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4190100c30 0x7f41901a5d90 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f41901065a0 0x7f41901006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 --2- 172.21.15.73:0/224866738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f4190068740 0x7f41901001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 >> 172.21.15.73:0/224866738 conn(0x7f41900772f0 msgr2=0x7f41900fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 shutdown_connections 2024-03-09T00:10:58.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.300+0000 7f4198644640 1 -- 172.21.15.73:0/224866738 wait complete. 2024-03-09T00:10:58.312 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:10:58.312 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:10:58.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:58 smithi073 ceph-mon[32309]: pgmap v126: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:58.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- 172.21.15.73:0/1896555240 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 msgr2=0x7f9d1c10d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- 172.21.15.73:0/1896555240 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c10d100 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 crypto rx=0x7f9d14009ca0 tx=0x7f9d1402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- 172.21.15.73:0/1896555240 shutdown_connections 2024-03-09T00:10:58.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- 172.21.15.73:0/1896555240 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c10d100 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- 172.21.15.73:0/1896555240 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c100e10 0x7f9d1c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- 172.21.15.73:0/1896555240 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- 172.21.15.73:0/1896555240 >> 172.21.15.73:0/1896555240 conn(0x7f9d1c0772f0 msgr2=0x7f9d1c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- 172.21.15.73:0/1896555240 shutdown_connections 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- 172.21.15.73:0/1896555240 wait complete. 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 Processor -- start 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- start start 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c1989d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c1994f0 0x7f9d1c1a3a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c1105d0 con 0x7f9d1c105530 2024-03-09T00:10:58.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c110450 con 0x7f9d1c1994f0 2024-03-09T00:10:58.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.419+0000 7f9d24122640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d1c110750 con 0x7f9d1c100460 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33790/0 (socket says 172.21.15.73:33790) 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 -- 172.21.15.73:0/647001167 learned_addr learned my addr 172.21.15.73:0/647001167 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d226bf640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c1994f0 0x7f9d1c1a3a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d21ebe640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c1989d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 -- 172.21.15.73:0/647001167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 msgr2=0x7f9d1c1989d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c1989d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 -- 172.21.15.73:0/647001167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c1994f0 msgr2=0x7f9d1c1a3a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c1994f0 0x7f9d1c1a3a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 -- 172.21.15.73:0/647001167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c1a4270 con 0x7f9d1c105530 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d21ebe640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d216bd640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800d770 tx=0x7f9d1800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d18017070 con 0x7f9d1c105530 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d18009c30 con 0x7f9d1c105530 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d1c1a4560 con 0x7f9d1c105530 2024-03-09T00:10:58.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d1800f620 con 0x7f9d1c105530 2024-03-09T00:10:58.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.420+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d1c1a4ab0 con 0x7f9d1c105530 2024-03-09T00:10:58.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.421+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d1c0680e0 con 0x7f9d1c105530 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.422+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9d1801a070 con 0x7f9d1c105530 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.423+0000 7f9d0effd640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 0x7f9d080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.423+0000 7f9d21ebe640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 0x7f9d080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.423+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9d180cec90 con 0x7f9d1c105530 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.424+0000 7f9d21ebe640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 0x7f9d080a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d100058c0 tx=0x7f9d10005d30 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.427+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9d18009de0 con 0x7f9d1c105530 2024-03-09T00:10:58.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:58 smithi102 ceph-mon[36106]: pgmap v126: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:58.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:58 smithi179 ceph-mon[36637]: pgmap v126: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:10:58.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.638+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d1c1087c0 con 0x7f9d1c105530 2024-03-09T00:10:58.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.638+0000 7f9d0effd640 1 -- 172.21.15.73:0/647001167 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f9d1809b660 con 0x7f9d1c105530 2024-03-09T00:10:58.634 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:10:58.635 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:10:58.635 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:10:58.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 msgr2=0x7f9d080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 0x7f9d080a4e30 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f9d100058c0 tx=0x7f9d10005d30 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 msgr2=0x7f9d1c198f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 crypto rx=0x7f9d1800d770 tx=0x7f9d1800dc30 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 shutdown_connections 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1c1994f0 0x7f9d1c1a3a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d080a2970 0x7f9d080a4e30 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1c105530 0x7f9d1c198f10 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 --2- 172.21.15.73:0/647001167 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d1c100460 0x7f9d1c1989d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.640+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 >> 172.21.15.73:0/647001167 conn(0x7f9d1c0772f0 msgr2=0x7f9d1c0fe2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.641+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 shutdown_connections 2024-03-09T00:10:58.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.641+0000 7f9d24122640 1 -- 172.21.15.73:0/647001167 wait complete. 2024-03-09T00:10:58.647 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:10:58.647 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:10:58.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.755+0000 7f360ec8c640 1 -- 172.21.15.73:0/2790513025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36081065b0 msgr2=0x7f360810e180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.755+0000 7f360ec8c640 1 --2- 172.21.15.73:0/2790513025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36081065b0 0x7f360810e180 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 crypto rx=0x7f3600009ca0 tx=0x7f360002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- 172.21.15.73:0/2790513025 shutdown_connections 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- 172.21.15.73:0/2790513025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f36081065b0 0x7f360810e180 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- 172.21.15.73:0/2790513025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3608101e90 0x7f3608105ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- 172.21.15.73:0/2790513025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 0x7f36081018c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- 172.21.15.73:0/2790513025 >> 172.21.15.73:0/2790513025 conn(0x7f36080fcda0 msgr2=0x7f36080ff1c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- 172.21.15.73:0/2790513025 shutdown_connections 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- 172.21.15.73:0/2790513025 wait complete. 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 Processor -- start 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- start start 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 0x7f360819d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 0x7f360819dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3608111650 con 0x7f3608197970 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081114d0 con 0x7f36081065b0 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360ec8c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081117d0 con 0x7f36081014e0 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360d229640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360d229640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52300/0 (socket says 172.21.15.73:52300) 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.756+0000 7f360d229640 1 -- 172.21.15.73:0/3937091664 learned_addr learned my addr 172.21.15.73:0/3937091664 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f3607fff640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 0x7f360819dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360ca28640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 0x7f360819d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 -- 172.21.15.73:0/3937091664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 msgr2=0x7f360819d7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 0x7f360819d7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 -- 172.21.15.73:0/3937091664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 msgr2=0x7f360819dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 0x7f360819dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 -- 172.21.15.73:0/3937091664 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3608198490 con 0x7f3608197970 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f3607fff640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 0x7f360819dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360d229640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f360002c700 tx=0x7f3600032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f3605ffb640 1 -- 172.21.15.73:0/3937091664 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600041020 con 0x7f3608197970 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f3605ffb640 1 -- 172.21.15.73:0/3937091664 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3600034cd0 con 0x7f3608197970 2024-03-09T00:10:58.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3608198720 con 0x7f3608197970 2024-03-09T00:10:58.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f3605ffb640 1 -- 172.21.15.73:0/3937091664 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3600039040 con 0x7f3608197970 2024-03-09T00:10:58.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.757+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36081a4910 con 0x7f3608197970 2024-03-09T00:10:58.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3605ffb640 1 -- 172.21.15.73:0/3937091664 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3600044020 con 0x7f3608197970 2024-03-09T00:10:58.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 Processor -- start 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 -- start start 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bedb570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8beddea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8be95360 con 0x55fb8becdfb0 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8be31ec0 con 0x55fb8bedbab0 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3611861740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8bddad30 con 0x55fb8bece960 2024-03-09T00:10:58.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52314/0 (socket says 172.21.15.73:52314) 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.760+0000 7f3607fff640 1 -- 172.21.15.73:0/4030776648 learned_addr learned my addr 172.21.15.73:0/4030776648 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3605ffb640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f360d229640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8beddea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f360ca28640 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bedb570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 -- 172.21.15.73:0/4030776648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 msgr2=0x55fb8bedb570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bedb570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 -- 172.21.15.73:0/4030776648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 msgr2=0x55fb8beddea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8beddea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 -- 172.21.15.73:0/4030776648 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb8bdd9bc0 con 0x55fb8becdfb0 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3605ffb640 1 -- 172.21.15.73:0/3937091664 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f36000f0ff0 con 0x7f3608197970 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f3607fff640 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f35fc005f60 tx=0x7f35fc009700 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be75a4b5c14d8f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f360d229640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 0x7f35f00a4e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36000ee570 tx=0x7f36000095c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f35eeffd640 1 -- 172.21.15.73:0/4030776648 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004030 con 0x55fb8becdfb0 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f35eeffd640 1 -- 172.21.15.73:0/4030776648 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35fc0041d0 con 0x55fb8becdfb0 2024-03-09T00:10:58.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.761+0000 7f35eeffd640 1 -- 172.21.15.73:0/4030776648 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc003e60 con 0x55fb8becdfb0 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- 172.21.15.73:0/4030776648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 msgr2=0x55fb8bece390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 crypto rx=0x7f35fc005f60 tx=0x7f35fc009700 comp rx=0 tx=0).stop 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- 172.21.15.73:0/4030776648 shutdown_connections 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8beddea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bedb570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 --2- 172.21.15.73:0/4030776648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bece390 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- 172.21.15.73:0/4030776648 >> 172.21.15.73:0/4030776648 conn(0x55fb8bec88d0 msgr2=0x55fb8bec8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- 172.21.15.73:0/4030776648 shutdown_connections 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- 172.21.15.73:0/4030776648 wait complete. 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 -- start start 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.762+0000 7f3611861740 1 Processor -- start 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8becdfb0 0x55fb8bed8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bece960 0x55fb8bed8fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8bdd04a0 con 0x55fb8bedbab0 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8be2df60 con 0x55fb8bece960 2024-03-09T00:10:58.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fb8bede870 con 0x55fb8becdfb0 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 10 client.0 _ll_register_callbacks cb 0x55fb8be2bd60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52322/0 (socket says 172.21.15.73:52322) 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 -- 172.21.15.73:0/1162733700 learned_addr learned my addr 172.21.15.73:0/1162733700 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8becdfb0 0x55fb8bed8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f360ca28640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bece960 0x55fb8bed8fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 -- 172.21.15.73:0/1162733700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8becdfb0 msgr2=0x55fb8bed8aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8becdfb0 0x55fb8bed8aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 -- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bece960 msgr2=0x55fb8bed8fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bece960 0x55fb8bed8fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 -- 172.21.15.73:0/1162733700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb8bb463f0 con 0x55fb8bedbab0 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3607fff640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f35fc013f40 tx=0x7f35fc003ca0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbf726feee69246b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f35e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc003960 con 0x55fb8bedbab0 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f35fc0130d0 con 0x55fb8bedbab0 2024-03-09T00:10:58.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35fc0041d0 con 0x55fb8bedbab0 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.763+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35fc011470 con 0x55fb8bedbab0 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 10 client.16371 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fb8bc107d0 con 0x55fb8bedbab0 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35fc0041d0 con 0x55fb8bedbab0 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f35fc02e020 con 0x55fb8bedbab0 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 10 client.16371 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 10 client.16371 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:58.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 20 client.16371 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8beebb00 0x55fb8beedef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 4 client.16371 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fb8be3f400 con 0x55fb8beebb00 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.764+0000 7f360ca28640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8beebb00 0x55fb8beedef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f35e7fff640 10 client.16371 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f360ca28640 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8beebb00 0x55fb8beedef0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=25bfabe1fe5cff59 server_cookie=4b6b6a44446ed79d in_seq=0 out_seq=0 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f35e7fff640 1 -- 172.21.15.73:0/1162733700 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fb8be3f400 con 0x55fb8beebb00 2024-03-09T00:10:58.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f35e7fff640 10 client.16371 handle_command_reply: tid=0 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 client.16371 shutdown 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 msgr2=0x55fb8bed9520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 crypto rx=0x7f35fc013f40 tx=0x7f35fc003ca0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 shutdown_connections 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8beebb00 0x55fb8beedef0 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed9520 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bece960 0x55fb8bed8fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f35e7fff640 0 client.16371 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 --2- 172.21.15.73:0/1162733700 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8becdfb0 0x55fb8bed8aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.765+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 >> 172.21.15.73:0/1162733700 conn(0x55fb8bec88d0 msgr2=0x55fb8beca620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.766+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 shutdown_connections 2024-03-09T00:10:58.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.766+0000 7f3611861740 1 -- 172.21.15.73:0/1162733700 wait complete. 2024-03-09T00:10:58.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.766+0000 7f3611861740 20 client.16371 trim_cache size 0 max 16384 2024-03-09T00:10:58.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 Processor -- start 2024-03-09T00:10:58.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 -- start start 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8becdfb0 0x55fb8bedb1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bed7a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081aa560 con 0x55fb8bedbab0 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081acb20 con 0x55fb8becdfb0 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3611861740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081aab60 con 0x55fb8bece960 2024-03-09T00:10:58.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52324/0 (socket says 172.21.15.73:52324) 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.777+0000 7f3607fff640 1 -- 172.21.15.73:0/3134187715 learned_addr learned my addr 172.21.15.73:0/3134187715 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f360ca28640 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bed7a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 -- 172.21.15.73:0/3134187715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 msgr2=0x55fb8bed7a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bed7a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 -- 172.21.15.73:0/3134187715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8becdfb0 msgr2=0x55fb8bedb1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8becdfb0 0x55fb8bedb1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 -- 172.21.15.73:0/3134187715 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fb8bec2da0 con 0x55fb8bedbab0 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3607fff640 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009e20 tx=0x7f35fc009550 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58a2e044f7d98476 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f35ee7fc640 1 -- 172.21.15.73:0/3134187715 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc004150 con 0x55fb8bedbab0 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f35ee7fc640 1 -- 172.21.15.73:0/3134187715 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35fc005550 con 0x55fb8bedbab0 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f35ee7fc640 1 -- 172.21.15.73:0/3134187715 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc00f480 con 0x55fb8bedbab0 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3611861740 1 -- 172.21.15.73:0/3134187715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 msgr2=0x55fb8bed5d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3611861740 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 secure :-1 s=READY pgs=742 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009e20 tx=0x7f35fc009550 comp rx=0 tx=0).stop 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.778+0000 7f3611861740 1 -- 172.21.15.73:0/3134187715 shutdown_connections 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8bedbab0 0x55fb8bed5d00 unknown :-1 s=CLOSED pgs=742 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bed7a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- 172.21.15.73:0/3134187715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8becdfb0 0x55fb8bedb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- 172.21.15.73:0/3134187715 >> 172.21.15.73:0/3134187715 conn(0x55fb8bec88d0 msgr2=0x55fb8bed7370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- 172.21.15.73:0/3134187715 shutdown_connections 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- 172.21.15.73:0/3134187715 wait complete. 2024-03-09T00:10:58.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- start start 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 Processor -- start 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bee10f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8bee1630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081ab1a0 con 0x55fb8becdfb0 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081ac4c0 con 0x55fb8bedbab0 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36081ab800 con 0x55fb8bece960 2024-03-09T00:10:58.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3611861740 10 client.0 _ll_register_callbacks cb 0x7f35fc009610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3607fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52328/0 (socket says 172.21.15.73:52328) 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f3607fff640 1 -- 172.21.15.73:0/1457407052 learned_addr learned my addr 172.21.15.73:0/1457407052 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.779+0000 7f360ca28640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bee10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8bee1630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 -- 172.21.15.73:0/1457407052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 msgr2=0x55fb8bee10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bee10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 -- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 msgr2=0x55fb8bee1630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8bee1630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 -- 172.21.15.73:0/1457407052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fb8bee1ca0 con 0x55fb8becdfb0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3607fff640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009ca0 tx=0x7f35fc0058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb4e0da3b58dfcca server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35fc009770 con 0x55fb8becdfb0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f35fc003960 con 0x55fb8becdfb0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35fc005550 con 0x55fb8becdfb0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35fc003b60 con 0x55fb8becdfb0 2024-03-09T00:10:58.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 10 client.16377 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fb8bf42dd0 con 0x55fb8becdfb0 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35fc005550 con 0x55fb8becdfb0 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f35fc02e070 con 0x55fb8becdfb0 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 10 client.16377 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 10 client.16377 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 20 client.16377 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8bf431a0 0x55fb8bf45590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:58.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 4 client.16377 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.780+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55fb8be39750 con 0x55fb8bf431a0 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f360ca28640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8bf431a0 0x55fb8bf45590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f35e7fff640 10 client.16377 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f360ca28640 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8bf431a0 0x55fb8bf45590 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8ee6d98df010eb2c server_cookie=4a7ae6726b748798 in_seq=0 out_seq=0 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f35e7fff640 1 -- 172.21.15.73:0/1457407052 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55fb8be39750 con 0x55fb8bf431a0 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f35e7fff640 10 client.16377 handle_command_reply: tid=0 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 client.16377 shutdown 2024-03-09T00:10:58.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 msgr2=0x55fb8bee2cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 crypto rx=0x7f35fc009ca0 tx=0x7f35fc0058b0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 shutdown_connections 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fb8bf431a0 0x55fb8bf45590 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fb8bedbab0 0x55fb8bee1630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fb8bece960 0x55fb8bee10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 --2- 172.21.15.73:0/1457407052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fb8becdfb0 0x55fb8bee2cb0 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f35e7fff640 0 client.16377 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.781+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 >> 172.21.15.73:0/1457407052 conn(0x55fb8bec88d0 msgr2=0x55fb8bed8a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 shutdown_connections 2024-03-09T00:10:58.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f3611861740 1 -- 172.21.15.73:0/1457407052 wait complete. 2024-03-09T00:10:58.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f3611861740 20 client.16377 trim_cache size 0 max 16384 2024-03-09T00:10:58.792 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:10:58.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 msgr2=0x7f35f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 0x7f35f00a4e30 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f36000ee570 tx=0x7f36000095c0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 msgr2=0x7f3608197e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 crypto rx=0x7f360002c700 tx=0x7f3600032f20 comp rx=0 tx=0).stop 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 shutdown_connections 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f35f00a2970 0x7f35f00a4e30 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3608197970 0x7f3608197e20 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f36081065b0 0x7f360819dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 --2- 172.21.15.73:0/3937091664 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f36081014e0 0x7f360819d7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:58.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 >> 172.21.15.73:0/3937091664 conn(0x7f36080fcda0 msgr2=0x7f36080ff0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:58.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 shutdown_connections 2024-03-09T00:10:58.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:58.782+0000 7f360ec8c640 1 -- 172.21.15.73:0/3937091664 wait complete. 2024-03-09T00:10:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:59 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/647001167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:10:59 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:59 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/647001167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:10:59 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:59 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/647001167' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:10:59 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:10:59.796 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:10:59.796 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:10:59.796 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:10:59.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 -- 172.21.15.73:0/1496166673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1014f0 msgr2=0x7f971c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 --2- 172.21.15.73:0/1496166673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1014f0 0x7f971c1018d0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 crypto rx=0x7f9718009900 tx=0x7f971802bc30 comp rx=0 tx=0).stop 2024-03-09T00:10:59.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 -- 172.21.15.73:0/1496166673 shutdown_connections 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 --2- 172.21.15.73:0/1496166673 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c1065c0 0x7f971c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 --2- 172.21.15.73:0/1496166673 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c101ea0 0x7f971c105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 --2- 172.21.15.73:0/1496166673 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1014f0 0x7f971c1018d0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 -- 172.21.15.73:0/1496166673 >> 172.21.15.73:0/1496166673 conn(0x7f971c0786f0 msgr2=0x7f971c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 -- 172.21.15.73:0/1496166673 shutdown_connections 2024-03-09T00:10:59.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.907+0000 7f9723b97640 1 -- 172.21.15.73:0/1496166673 wait complete. 2024-03-09T00:10:59.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 Processor -- start 2024-03-09T00:10:59.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 -- start start 2024-03-09T00:10:59.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c1014f0 0x7f971c19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c197950 0x7f971c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c111650 con 0x7f971c1065c0 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1114d0 con 0x7f971c1014f0 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9723b97640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1117d0 con 0x7f971c197950 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52336/0 (socket says 172.21.15.73:52336) 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 -- 172.21.15.73:0/1070360140 learned_addr learned my addr 172.21.15.73:0/1070360140 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9722134640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c197950 0x7f971c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721933640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c1014f0 0x7f971c19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 -- 172.21.15.73:0/1070360140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c197950 msgr2=0x7f971c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c197950 0x7f971c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 -- 172.21.15.73:0/1070360140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c1014f0 msgr2=0x7f971c19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c1014f0 0x7f971c19d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.908+0000 7f9721132640 1 -- 172.21.15.73:0/1070360140 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f971c198470 con 0x7f971c1065c0 2024-03-09T00:10:59.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9721132640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f970c00e9c0 tx=0x7f970c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9712ffd640 1 -- 172.21.15.73:0/1070360140 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c00cd80 con 0x7f971c1065c0 2024-03-09T00:10:59.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9712ffd640 1 -- 172.21.15.73:0/1070360140 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f970c005500 con 0x7f971c1065c0 2024-03-09T00:10:59.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f971c198760 con 0x7f971c1065c0 2024-03-09T00:10:59.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9712ffd640 1 -- 172.21.15.73:0/1070360140 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c010690 con 0x7f971c1065c0 2024-03-09T00:10:59.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.909+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f971c1a4a00 con 0x7f971c1065c0 2024-03-09T00:10:59.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.911+0000 7f9712ffd640 1 -- 172.21.15.73:0/1070360140 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f970c0046c0 con 0x7f971c1065c0 2024-03-09T00:10:59.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.911+0000 7f972677a740 1 Processor -- start 2024-03-09T00:10:59.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 -- start start 2024-03-09T00:10:59.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9712ffd640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 0x7f97080a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f5398d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f52f3b40 con 0x5636f538fd80 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f529cd80 con 0x5636f5399280 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f972677a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f52efbe0 con 0x5636f53906a0 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.922 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9721933640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 0x7f97080a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52350/0 (socket says 172.21.15.73:52350) 2024-03-09T00:10:59.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9712ffd640 1 -- 172.21.15.73:0/1070360140 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f970c0cf4e0 con 0x7f971c1065c0 2024-03-09T00:10:59.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 -- 172.21.15.73:0/4284192071 learned_addr learned my addr 172.21.15.73:0/4284192071 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:59.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.923 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9721132640 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f5398d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 -- 172.21.15.73:0/4284192071 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 msgr2=0x5636f5398d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f5398d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 -- 172.21.15.73:0/4284192071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 msgr2=0x5636f539b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 -- 172.21.15.73:0/4284192071 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636f50d07e0 con 0x5636f538fd80 2024-03-09T00:10:59.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9721933640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 0x7f97080a4eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9718009230 tx=0x7f9718005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.912+0000 7f9722134640 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9714013ad0 tx=0x7f9714005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4c2c5b0ab76a0544 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f9707fff640 1 -- 172.21.15.73:0/4284192071 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714013130 con 0x5636f538fd80 2024-03-09T00:10:59.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f9707fff640 1 -- 172.21.15.73:0/4284192071 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f971400f940 con 0x5636f538fd80 2024-03-09T00:10:59.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f9707fff640 1 -- 172.21.15.73:0/4284192071 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9714005310 con 0x5636f538fd80 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 -- 172.21.15.73:0/4284192071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 msgr2=0x5636f5390160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 crypto rx=0x7f9714013ad0 tx=0x7f9714005bd0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 -- 172.21.15.73:0/4284192071 shutdown_connections 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f5398d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 --2- 172.21.15.73:0/4284192071 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5390160 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.926 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 -- 172.21.15.73:0/4284192071 >> 172.21.15.73:0/4284192071 conn(0x5636f538a4b0 msgr2=0x5636f538a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 -- 172.21.15.73:0/4284192071 shutdown_connections 2024-03-09T00:10:59.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.913+0000 7f972677a740 1 -- 172.21.15.73:0/4284192071 wait complete. 2024-03-09T00:10:59.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 -- start start 2024-03-09T00:10:59.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 Processor -- start 2024-03-09T00:10:59.927 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f538fd80 0x5636f5398ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f5396640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f539c010 con 0x5636f53906a0 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f52b01c0 con 0x5636f5399280 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5636f539c190 con 0x5636f538fd80 2024-03-09T00:10:59.928 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f972677a740 10 client.0 _ll_register_callbacks cb 0x5636f529f7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:59.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721132640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721132640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52364/0 (socket says 172.21.15.73:52364) 2024-03-09T00:10:59.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721933640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f538fd80 0x5636f5398ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721132640 1 -- 172.21.15.73:0/3906877996 learned_addr learned my addr 172.21.15.73:0/3906877996 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:59.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9722134640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f5396640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721132640 1 -- 172.21.15.73:0/3906877996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f538fd80 msgr2=0x5636f5398ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.914+0000 7f9721132640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f538fd80 0x5636f5398ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f9721132640 1 -- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 msgr2=0x5636f5396640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f9721132640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f5396640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f9721132640 1 -- 172.21.15.73:0/3906877996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636f5130f70 con 0x5636f53906a0 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f9722134640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f5396640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f9721132640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f970c097c10 tx=0x7f970c09b2f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74787981d102654b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f970c09beb0 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f970c0ccb20 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f971400f940 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f970c09b7b0 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 10 client.16389 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5636f4fc8170 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f971400f940 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f970c097e20 con 0x5636f53906a0 2024-03-09T00:10:59.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 10 client.16389 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 10 client.16389 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 20 client.16389 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f53a9360 0x5636f53ab750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 4 client.16389 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.915+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5636f52c9960 con 0x5636f53a9360 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.916+0000 7f9721933640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f53a9360 0x5636f53ab750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.916+0000 7f97057fa640 10 client.16389 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:59.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.916+0000 7f9721933640 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f53a9360 0x5636f53ab750 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=53909dd7a894431b server_cookie=da922bda7ab4ed59 in_seq=0 out_seq=0 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f97057fa640 1 -- 172.21.15.73:0/3906877996 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5636f52c9960 con 0x5636f53a9360 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f97057fa640 10 client.16389 handle_command_reply: tid=0 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 client.16389 shutdown 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 msgr2=0x5636f5396030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 crypto rx=0x7f970c097c10 tx=0x7f970c09b2f0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 shutdown_connections 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f53a9360 0x5636f53ab750 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f5396640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f53906a0 0x5636f5396030 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f97057fa640 0 client.16389 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 --2- 172.21.15.73:0/3906877996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f538fd80 0x5636f5398ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 >> 172.21.15.73:0/3906877996 conn(0x5636f538a4b0 msgr2=0x5636f538c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 shutdown_connections 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 1 -- 172.21.15.73:0/3906877996 wait complete. 2024-03-09T00:10:59.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.917+0000 7f972677a740 20 client.16389 trim_cache size 0 max 16384 2024-03-09T00:10:59.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 Processor -- start 2024-03-09T00:10:59.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 -- start start 2024-03-09T00:10:59.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 0x5636f5398760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.941 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f5399280 0x5636f5398ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ac780 con 0x5636f538fd80 2024-03-09T00:10:59.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1aed40 con 0x5636f53906a0 2024-03-09T00:10:59.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f972677a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1acd80 con 0x5636f5399280 2024-03-09T00:10:59.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.942 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52372/0 (socket says 172.21.15.73:52372) 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 -- 172.21.15.73:0/3554146608 learned_addr learned my addr 172.21.15.73:0/3554146608 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9722134640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f5399280 0x5636f5398ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721132640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 0x5636f5398760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 -- 172.21.15.73:0/3554146608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f5399280 msgr2=0x5636f5398ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f5399280 0x5636f5398ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 -- 172.21.15.73:0/3554146608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 msgr2=0x5636f5398760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.943 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 0x5636f5398760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721933640 1 -- 172.21.15.73:0/3554146608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5636f53969b0 con 0x5636f538fd80 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.932+0000 7f9721132640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 0x5636f5398760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f9721933640 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f971802c170 tx=0x7f971803f000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc53749424cb8563 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f97077fe640 1 -- 172.21.15.73:0/3554146608 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f971802cb40 con 0x5636f538fd80 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f97077fe640 1 -- 172.21.15.73:0/3554146608 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f971802cce0 con 0x5636f538fd80 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f97077fe640 1 -- 172.21.15.73:0/3554146608 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718051250 con 0x5636f538fd80 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 -- 172.21.15.73:0/3554146608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 msgr2=0x5636f5395ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.944 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 secure :-1 s=READY pgs=748 cs=0 l=1 rev1=1 crypto rx=0x7f971802c170 tx=0x7f971803f000 comp rx=0 tx=0).stop 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 -- 172.21.15.73:0/3554146608 shutdown_connections 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f5399280 0x5636f5398ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f53906a0 0x5636f5398760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 --2- 172.21.15.73:0/3554146608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f5395ff0 unknown :-1 s=CLOSED pgs=748 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 -- 172.21.15.73:0/3554146608 >> 172.21.15.73:0/3554146608 conn(0x5636f538a4b0 msgr2=0x5636f5391320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 -- 172.21.15.73:0/3554146608 shutdown_connections 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.933+0000 7f972677a740 1 -- 172.21.15.73:0/3554146608 wait complete. 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 -- start start 2024-03-09T00:10:59.945 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 Processor -- start 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f539d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539dd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ad3c0 con 0x5636f538fd80 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ae6e0 con 0x5636f5399280 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f971c1ada20 con 0x5636f53906a0 2024-03-09T00:10:59.946 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f972677a740 10 client.0 _ll_register_callbacks cb 0x7f9714009b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52374/0 (socket says 172.21.15.73:52374) 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 -- 172.21.15.73:0/1190921234 learned_addr learned my addr 172.21.15.73:0/1190921234 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721132640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f539d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 -- 172.21.15.73:0/1190921234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 msgr2=0x5636f539d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f539d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 -- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 msgr2=0x5636f539dd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:10:59.947 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.934+0000 7f9721933640 1 -- 172.21.15.73:0/1190921234 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5636f53abac0 con 0x5636f538fd80 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f9721933640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f97180046c0 tx=0x7f9718005e50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d5a01024a895474 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9718004240 con 0x5636f538fd80 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f97180043e0 con 0x5636f538fd80 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5636f52b5080 con 0x5636f538fd80 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9718051220 con 0x5636f538fd80 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 10 client.16395 fetch_fsmap learned FSMap version 20 2024-03-09T00:10:59.948 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5636f54012f0 con 0x5636f538fd80 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5636f52b5080 con 0x5636f538fd80 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f97180035f0 con 0x5636f538fd80 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 10 client.16395 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 10 client.16395 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 20 client.16395 populate_metadata read hostname 'smithi073' 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f54019c0 0x5636f5403de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 4 client.16395 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:10:59.949 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5636f5301080 con 0x5636f54019c0 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.935+0000 7f9721132640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f54019c0 0x5636f5403de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.936+0000 7f97057fa640 10 client.16395 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.936+0000 7f9721132640 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f54019c0 0x5636f5403de0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=917d03f5072be499 server_cookie=8d4d1f794f68a21b in_seq=0 out_seq=0 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.936+0000 7f97057fa640 1 -- 172.21.15.73:0/1190921234 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5636f5301080 con 0x5636f54019c0 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.936+0000 7f97057fa640 10 client.16395 handle_command_reply: tid=0 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.936+0000 7f972677a740 1 client.16395 shutdown 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 msgr2=0x5636f53a0450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.950 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 crypto rx=0x7f97180046c0 tx=0x7f9718005e50 comp rx=0 tx=0).stop 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 shutdown_connections 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5636f54019c0 0x5636f5403de0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5636f5399280 0x5636f539dd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5636f53906a0 0x5636f539d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 --2- 172.21.15.73:0/1190921234 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5636f538fd80 0x5636f53a0450 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 >> 172.21.15.73:0/1190921234 conn(0x5636f538a4b0 msgr2=0x5636f5388bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f97057fa640 0 client.16395 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:10:59.951 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 shutdown_connections 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 1 -- 172.21.15.73:0/1190921234 wait complete. 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f972677a740 20 client.16395 trim_cache size 0 max 16384 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "e039d235-8756-4059-be01-c17f52c28679", 2024-03-09T00:10:59.952 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:10:59.953 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:10:59.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 msgr2=0x7f97080a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 0x7f97080a4eb0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f9718009230 tx=0x7f9718005fd0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 msgr2=0x7f971c19dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 secure :-1 s=READY pgs=745 cs=0 l=1 rev1=1 crypto rx=0x7f970c00e9c0 tx=0x7f970c00ee80 comp rx=0 tx=0).stop 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 shutdown_connections 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f971c197950 0x7f971c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f97080a29f0 0x7f97080a4eb0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f971c1065c0 0x7f971c19dce0 unknown :-1 s=CLOSED pgs=745 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 --2- 172.21.15.73:0/1070360140 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f971c1014f0 0x7f971c19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:10:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.937+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 >> 172.21.15.73:0/1070360140 conn(0x7f971c0786f0 msgr2=0x7f971c0ff4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:10:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.938+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 shutdown_connections 2024-03-09T00:10:59.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:10:59.938+0000 7f9723b97640 1 -- 172.21.15.73:0/1070360140 wait complete. 2024-03-09T00:10:59.956 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:10:59.956 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:11:00.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 -- 172.21.15.73:0/1499453754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 msgr2=0x7f93b01018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 --2- 172.21.15.73:0/1499453754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b01018d0 secure :-1 s=READY pgs=750 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009900 tx=0x7f93a002bc30 comp rx=0 tx=0).stop 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 -- 172.21.15.73:0/1499453754 shutdown_connections 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 --2- 172.21.15.73:0/1499453754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b01065c0 0x7f93b010e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 --2- 172.21.15.73:0/1499453754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b0101ea0 0x7f93b0105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 --2- 172.21.15.73:0/1499453754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b01018d0 unknown :-1 s=CLOSED pgs=750 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 -- 172.21.15.73:0/1499453754 >> 172.21.15.73:0/1499453754 conn(0x7f93b00786b0 msgr2=0x7f93b0078ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 -- 172.21.15.73:0/1499453754 shutdown_connections 2024-03-09T00:11:00.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.073+0000 7f93b6129640 1 -- 172.21.15.73:0/1499453754 wait complete. 2024-03-09T00:11:00.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 Processor -- start 2024-03-09T00:11:00.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 -- start start 2024-03-09T00:11:00.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b01065c0 0x7f93b019dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b0197a10 0x7f93b0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b0111670 con 0x7f93b01014f0 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01114f0 con 0x7f93b0197a10 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93b6129640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01117f0 con 0x7f93b01065c0 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52382/0 (socket says 172.21.15.73:52382) 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93af7fe640 1 -- 172.21.15.73:0/1035194916 learned_addr learned my addr 172.21.15.73:0/1035194916 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93aeffd640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b01065c0 0x7f93b019dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.074+0000 7f93affff640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b0197a10 0x7f93b0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 -- 172.21.15.73:0/1035194916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b01065c0 msgr2=0x7f93b019dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b01065c0 0x7f93b019dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 -- 172.21.15.73:0/1035194916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b0197a10 msgr2=0x7f93b0197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b0197a10 0x7f93b0197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 -- 172.21.15.73:0/1035194916 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b0198530 con 0x7f93b01014f0 2024-03-09T00:11:00.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93af7fe640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f93a0006010 tx=0x7f93a002cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93acff9640 1 -- 172.21.15.73:0/1035194916 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0041070 con 0x7f93b01014f0 2024-03-09T00:11:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b01987c0 con 0x7f93b01014f0 2024-03-09T00:11:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93acff9640 1 -- 172.21.15.73:0/1035194916 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a0004450 con 0x7f93b01014f0 2024-03-09T00:11:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93acff9640 1 -- 172.21.15.73:0/1035194916 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0003e00 con 0x7f93b01014f0 2024-03-09T00:11:00.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.075+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01a4a10 con 0x7f93b01014f0 2024-03-09T00:11:00.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.077+0000 7f93acff9640 1 -- 172.21.15.73:0/1035194916 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f93a0044070 con 0x7f93b01014f0 2024-03-09T00:11:00.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93acff9640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 0x7f93980a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 Processor -- start 2024-03-09T00:11:00.082 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93aeffd640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 0x7f93980a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 -- start start 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93acff9640 1 -- 172.21.15.73:0/1035194916 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f93a00f0b10 con 0x7f93b01014f0 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 0x55588d7fa720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 0x55588d7fd050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d754e00 con 0x55588d7f2080 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d6fdb10 con 0x55588d7fac60 2024-03-09T00:11:00.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.078+0000 7f93b8de1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d750ea0 con 0x55588d7f1760 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:56322/0 (socket says 172.21.15.73:56322) 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 -- 172.21.15.73:0/80155148 learned_addr learned my addr 172.21.15.73:0/80155148 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93aeffd640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 0x55588d7fa720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93affff640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 0x55588d7fd050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93aeffd640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 0x7f93980a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93b0100e30 tx=0x7f93a4005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 -- 172.21.15.73:0/80155148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 msgr2=0x55588d7fd050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 0x55588d7fd050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 -- 172.21.15.73:0/80155148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 msgr2=0x55588d7fa720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 0x55588d7fa720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 -- 172.21.15.73:0/80155148 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55588d5337d0 con 0x55588d7f1760 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93aeffd640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 0x55588d7fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93affff640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 0x55588d7fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f93af7fe640 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93a00bcf30 tx=0x7f93a00bcea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8f62b60f75b4ed9a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f9395ffb640 1 -- 172.21.15.73:0/80155148 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a004b040 con 0x55588d7f1760 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f9395ffb640 1 -- 172.21.15.73:0/80155148 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a00c1040 con 0x55588d7f1760 2024-03-09T00:11:00.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.079+0000 7f9395ffb640 1 -- 172.21.15.73:0/80155148 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a00b9610 con 0x55588d7f1760 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 -- 172.21.15.73:0/80155148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 msgr2=0x55588d7f1b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7f93a00bcf30 tx=0x7f93a00bcea0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 -- 172.21.15.73:0/80155148 shutdown_connections 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7fac60 0x55588d7fd050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7f2080 0x55588d7fa720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 --2- 172.21.15.73:0/80155148 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d7f1b40 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 -- 172.21.15.73:0/80155148 >> 172.21.15.73:0/80155148 conn(0x55588d7ebe90 msgr2=0x55588d7ec270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.086 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 -- 172.21.15.73:0/80155148 shutdown_connections 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.080+0000 7f93b8de1740 1 -- 172.21.15.73:0/80155148 wait complete. 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 -- start start 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 Processor -- start 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7f77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d7fd810 con 0x55588d7fac60 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d7110a0 con 0x55588d7f1760 2024-03-09T00:11:00.087 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55588d7fd990 con 0x55588d7f2080 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93b8de1740 10 client.0 _ll_register_callbacks cb 0x55588d710a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:56766/0 (socket says 172.21.15.73:56766) 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 -- 172.21.15.73:0/3451567555 learned_addr learned my addr 172.21.15.73:0/3451567555 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52394/0 (socket says 172.21.15.73:52394) 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93aeffd640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7f77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.088 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 -- 172.21.15.73:0/3451567555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 msgr2=0x55588d7f77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7f77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 -- 172.21.15.73:0/3451567555 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 msgr2=0x55588d7f7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.081+0000 7f93af7fe640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93af7fe640 1 -- 172.21.15.73:0/3451567555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55588d4693f0 con 0x55588d7f1760 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93affff640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93aeffd640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93af7fe640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f93a00bcca0 tx=0x7f93a00c5690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c01b25e56c53f61 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:11:00.089 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a004b040 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93a00c1040 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f93a00c5200 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93a00c5440 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93b8de1740 10 client.25739 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55588d4290f0 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a00c1040 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.082+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f93a00bbe60 con 0x55588d7f1760 2024-03-09T00:11:00.090 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 10 client.25739 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 10 client.25739 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 20 client.25739 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d80a2e0 0x55588d80c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 4 client.25739 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55588d762340 con 0x55588d80a2e0 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93aeffd640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d80a2e0 0x55588d80c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f938effd640 10 client.25739 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.083+0000 7f93aeffd640 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d80a2e0 0x55588d80c6d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c72f82fee3da2e4 server_cookie=a8e133498e5a2a2a in_seq=0 out_seq=0 2024-03-09T00:11:00.091 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f938effd640 1 -- 172.21.15.73:0/3451567555 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55588d762340 con 0x55588d80a2e0 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f938effd640 10 client.25739 handle_command_reply: tid=0 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 client.25739 shutdown 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 msgr2=0x55588d7fa450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f93a00bcca0 tx=0x7f93a00c5690 comp rx=0 tx=0).stop 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 shutdown_connections 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d80a2e0 0x55588d80c6d0 crc :-1 s=READY pgs=119 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d7f7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.092 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7f77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 --2- 172.21.15.73:0/3451567555 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7fa450 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 >> 172.21.15.73:0/3451567555 conn(0x55588d7ebe90 msgr2=0x55588d7edbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f938effd640 0 client.25739 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.084+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 shutdown_connections 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.085+0000 7f93b8de1740 1 -- 172.21.15.73:0/3451567555 wait complete. 2024-03-09T00:11:00.093 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.085+0000 7f93b8de1740 20 client.25739 trim_cache size 0 max 16384 2024-03-09T00:11:00.098 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.101+0000 7f93b8de1740 1 Processor -- start 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 -- start start 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7f9f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7fa450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01aa540 con 0x55588d7fac60 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01acb00 con 0x55588d7f1760 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93b8de1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01aab40 con 0x55588d7f2080 2024-03-09T00:11:00.099 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52406/0 (socket says 172.21.15.73:52406) 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93aeffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7fa450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 -- 172.21.15.73:0/299827882 learned_addr learned my addr 172.21.15.73:0/299827882 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93af7fe640 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7f9f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 -- 172.21.15.73:0/299827882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 msgr2=0x55588d7fa450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7fa450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 -- 172.21.15.73:0/299827882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 msgr2=0x55588d7f9f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7f9f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.100 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 -- 172.21.15.73:0/299827882 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55588d7f0850 con 0x55588d7fac60 2024-03-09T00:11:00.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93aeffd640 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:11:00.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.102+0000 7f93affff640 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f93a8003e70 tx=0x7f93a8004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7db686c6bde7ea5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93957fa640 1 -- 172.21.15.73:0/299827882 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8021070 con 0x55588d7fac60 2024-03-09T00:11:00.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93957fa640 1 -- 172.21.15.73:0/299827882 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a8011d60 con 0x55588d7fac60 2024-03-09T00:11:00.101 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93957fa640 1 -- 172.21.15.73:0/299827882 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a801b3c0 con 0x55588d7fac60 2024-03-09T00:11:00.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 -- 172.21.15.73:0/299827882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 msgr2=0x55588d801a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 secure :-1 s=READY pgs=752 cs=0 l=1 rev1=1 crypto rx=0x7f93a8003e70 tx=0x7f93a8004410 comp rx=0 tx=0).stop 2024-03-09T00:11:00.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 -- 172.21.15.73:0/299827882 shutdown_connections 2024-03-09T00:11:00.105 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d801a90 unknown :-1 s=CLOSED pgs=752 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f2080 0x55588d7fa450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 --2- 172.21.15.73:0/299827882 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f1760 0x55588d7f9f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 -- 172.21.15.73:0/299827882 >> 172.21.15.73:0/299827882 conn(0x55588d7ebe90 msgr2=0x55588d80c2f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 -- 172.21.15.73:0/299827882 shutdown_connections 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.103+0000 7f93b8de1740 1 -- 172.21.15.73:0/299827882 wait complete. 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 -- start start 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 Processor -- start 2024-03-09T00:11:00.106 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d801060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 0x55588d8015a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01ab180 con 0x55588d7fac60 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01ac4a0 con 0x55588d7f2080 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01ab7e0 con 0x55588d7f1760 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93b8de1740 10 client.0 _ll_register_callbacks cb 0x7f93a00bcae0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93affff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52414/0 (socket says 172.21.15.73:52414) 2024-03-09T00:11:00.107 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93affff640 1 -- 172.21.15.73:0/2394118801 learned_addr learned my addr 172.21.15.73:0/2394118801 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93af7fe640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d801060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.104+0000 7f93aeffd640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 0x55588d8015a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 -- 172.21.15.73:0/2394118801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 msgr2=0x55588d801060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d801060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 -- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 msgr2=0x55588d8015a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 0x55588d8015a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 -- 172.21.15.73:0/2394118801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55588d85fbb0 con 0x55588d7fac60 2024-03-09T00:11:00.108 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93aeffd640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 0x55588d8015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93affff640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f93a801f040 tx=0x7f93a801ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d07b2dc74c7e5a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f938effd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a802b020 con 0x55588d7fac60 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93a8011d60 con 0x55588d7fac60 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f93a8011680 con 0x55588d7fac60 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93a8036050 con 0x55588d7fac60 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93b8de1740 10 client.16413 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.105+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55588d860d60 con 0x55588d7fac60 2024-03-09T00:11:00.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a8011d60 con 0x55588d7fac60 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f93a8005520 con 0x55588d7fac60 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 10 client.16413 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 10 client.16413 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 20 client.16413 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d861320 0x55588d863740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 4 client.16413 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55588d75c690 con 0x55588d861320 2024-03-09T00:11:00.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93af7fe640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d861320 0x55588d863740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f93af7fe640 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d861320 0x55588d863740 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cdc467b6fbde10b server_cookie=715e4b2d85bbc02f in_seq=0 out_seq=0 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.106+0000 7f938effd640 10 client.16413 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f938effd640 1 -- 172.21.15.73:0/2394118801 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55588d75c690 con 0x55588d861320 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f938effd640 10 client.16413 handle_command_reply: tid=0 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 client.16413 shutdown 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 msgr2=0x55588d85f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 crypto rx=0x7f93a801f040 tx=0x7f93a801ce60 comp rx=0 tx=0).stop 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 shutdown_connections 2024-03-09T00:11:00.111 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55588d861320 0x55588d863740 crc :-1 s=READY pgs=120 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55588d7fac60 0x55588d85f670 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55588d7f2080 0x55588d8015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f938effd640 0 client.16413 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 --2- 172.21.15.73:0/2394118801 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55588d7f1760 0x55588d801060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 >> 172.21.15.73:0/2394118801 conn(0x55588d7ebe90 msgr2=0x55588d7f5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 shutdown_connections 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 1 -- 172.21.15.73:0/2394118801 wait complete. 2024-03-09T00:11:00.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.107+0000 7f93b8de1740 20 client.16413 trim_cache size 0 max 16384 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2092 inodes in the stack)", 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "e039d235-8756-4059-be01-c17f52c28679": { 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "e039d235-8756-4059-be01-c17f52c28679", 2024-03-09T00:11:00.113 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:11:00.114 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:11:00.114 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:11:00.114 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 msgr2=0x7f93980a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 0x7f93980a4e30 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f93b0100e30 tx=0x7f93a4005f90 comp rx=0 tx=0).stop 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 msgr2=0x7f93b019d860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 crypto rx=0x7f93a0006010 tx=0x7f93a002cb30 comp rx=0 tx=0).stop 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 shutdown_connections 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f93b0197a10 0x7f93b0197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f93b01065c0 0x7f93b019dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f93980a2970 0x7f93980a4e30 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 --2- 172.21.15.73:0/1035194916 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f93b01014f0 0x7f93b019d860 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:00.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 >> 172.21.15.73:0/1035194916 conn(0x7f93b00786b0 msgr2=0x7f93b00ff570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:00.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 shutdown_connections 2024-03-09T00:11:00.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:00.108+0000 7f93b6129640 1 -- 172.21.15.73:0/1035194916 wait complete. 2024-03-09T00:11:00.117 INFO:tasks.cephfs.filesystem:scrub status for tag:e039d235-8756-4059-be01-c17f52c28679 - {'path': '/', 'tag': 'e039d235-8756-4059-be01-c17f52c28679', 'options': 'recursive,force'} 2024-03-09T00:11:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: pgmap v127: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:11:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:11:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:11:00.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:11:00.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:00.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:00 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: pgmap v127: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:11:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:00.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:00 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:11:00.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: pgmap v127: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:00.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:11:00.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:11:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:11:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:11:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:00 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:11:01.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:01 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:01.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:01 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:01.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:01 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:02.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:02 smithi073 ceph-mon[32309]: pgmap v128: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:02.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:02 smithi102 ceph-mon[36106]: pgmap v128: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:02.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:02 smithi179 ceph-mon[36637]: pgmap v128: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:04 smithi073 ceph-mon[32309]: pgmap v129: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:04 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:11:04.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:04 smithi102 ceph-mon[36106]: pgmap v129: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:04 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:11:04.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:04 smithi179 ceph-mon[36637]: pgmap v129: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:04.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:04 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:11:06.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:06 smithi073 ceph-mon[32309]: pgmap v130: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:06.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:06 smithi102 ceph-mon[36106]: pgmap v130: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:06.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:06 smithi179 ceph-mon[36637]: pgmap v130: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:08.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:08 smithi073 ceph-mon[32309]: pgmap v131: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:08.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:08 smithi102 ceph-mon[36106]: pgmap v131: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:08.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:08 smithi179 ceph-mon[36637]: pgmap v131: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:09.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:09 smithi073 ceph-mon[32309]: pgmap v132: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:09.509 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:09 smithi102 ceph-mon[36106]: pgmap v132: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:09.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:09 smithi179 ceph-mon[36637]: pgmap v132: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:12.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:12 smithi073 ceph-mon[32309]: pgmap v133: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:12.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:12 smithi102 ceph-mon[36106]: pgmap v133: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:12.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:12 smithi179 ceph-mon[36637]: pgmap v133: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:11:14.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:14 smithi073 ceph-mon[32309]: pgmap v134: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:14.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:14 smithi102 ceph-mon[36106]: pgmap v134: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:14.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:14 smithi179 ceph-mon[36637]: pgmap v134: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:11:16.385 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:16 smithi179 ceph-mon[36637]: pgmap v135: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:11:16.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:16 smithi073 ceph-mon[32309]: pgmap v135: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:11:16.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:16 smithi102 ceph-mon[36106]: pgmap v135: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s rd, 170 B/s wr, 1 op/s 2024-03-09T00:11:18.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:18 smithi073 ceph-mon[32309]: pgmap v136: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-09T00:11:18.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:18 smithi102 ceph-mon[36106]: pgmap v136: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-09T00:11:18.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:18 smithi179 ceph-mon[36637]: pgmap v136: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.4 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-09T00:11:20.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:20 smithi073 ceph-mon[32309]: pgmap v137: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:20.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:20 smithi102 ceph-mon[36106]: pgmap v137: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:20.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:20 smithi179 ceph-mon[36637]: pgmap v137: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:22.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:22 smithi073 ceph-mon[32309]: pgmap v138: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:22.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:22 smithi102 ceph-mon[36106]: pgmap v138: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:22.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:22 smithi179 ceph-mon[36637]: pgmap v138: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:11:24.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:24 smithi073 ceph-mon[32309]: pgmap v139: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:11:24.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:24 smithi102 ceph-mon[36106]: pgmap v139: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:11:24.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:24 smithi179 ceph-mon[36637]: pgmap v139: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:11:26.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:26 smithi073 ceph-mon[32309]: pgmap v140: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:26.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:26 smithi102 ceph-mon[36106]: pgmap v140: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:26.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:26 smithi179 ceph-mon[36637]: pgmap v140: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:28.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:28 smithi073 ceph-mon[32309]: pgmap v141: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:28.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:28 smithi102 ceph-mon[36106]: pgmap v141: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:28.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:28 smithi179 ceph-mon[36637]: pgmap v141: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:30.118 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:11:30.118 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:11:30.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- 172.21.15.73:0/2311013899 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76240690f0 msgr2=0x7f7624105ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 --2- 172.21.15.73:0/2311013899 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76240690f0 0x7f7624105ec0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 crypto rx=0x7f7620009900 tx=0x7f762002bc30 comp rx=0 tx=0).stop 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- 172.21.15.73:0/2311013899 shutdown_connections 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 --2- 172.21.15.73:0/2311013899 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f76241065a0 0x7f762410e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 --2- 172.21.15.73:0/2311013899 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76240690f0 0x7f7624105ec0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 --2- 172.21.15.73:0/2311013899 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f7624068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- 172.21.15.73:0/2311013899 >> 172.21.15.73:0/2311013899 conn(0x7f76240772f0 msgr2=0x7f7624077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- 172.21.15.73:0/2311013899 shutdown_connections 2024-03-09T00:11:30.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- 172.21.15.73:0/2311013899 wait complete. 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 Processor -- start 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.237+0000 7f762bb8b640 1 -- start start 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f762419d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7624197890 0x7f7624197d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624111610 con 0x7f76241065a0 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624111490 con 0x7f7624197890 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7624111790 con 0x7f7624068740 2024-03-09T00:11:30.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39122/0 (socket says 172.21.15.73:39122) 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 -- 172.21.15.73:0/943682567 learned_addr learned my addr 172.21.15.73:0/943682567 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762a128640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7624197890 0x7f7624197d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629927640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f762419d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 -- 172.21.15.73:0/943682567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 msgr2=0x7f762419d6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f762419d6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 -- 172.21.15.73:0/943682567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7624197890 msgr2=0x7f7624197d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7624197890 0x7f7624197d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 -- 172.21.15.73:0/943682567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76241983b0 con 0x7f76241065a0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629927640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f762419d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7629126640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7620000c00 tx=0x7f762002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7616ffd640 1 -- 172.21.15.73:0/943682567 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620041070 con 0x7f76241065a0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7616ffd640 1 -- 172.21.15.73:0/943682567 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7620004450 con 0x7f76241065a0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7624198640 con 0x7f76241065a0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.238+0000 7f7616ffd640 1 -- 172.21.15.73:0/943682567 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7620034440 con 0x7f76241065a0 2024-03-09T00:11:30.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.239+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76241a4890 con 0x7f76241065a0 2024-03-09T00:11:30.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.241+0000 7f7616ffd640 1 -- 172.21.15.73:0/943682567 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7620044070 con 0x7f76241065a0 2024-03-09T00:11:30.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 Processor -- start 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 -- start start 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad90720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad93050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919aceae00 con 0x55919ad87760 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919ac93b10 con 0x55919ad88080 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762e768740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919ace6ea0 con 0x55919ad90c60 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762a128640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762a128640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39136/0 (socket says 172.21.15.73:39136) 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762a128640 1 -- 172.21.15.73:0/3118300271 learned_addr learned my addr 172.21.15.73:0/3118300271 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f7616ffd640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 0x7f76100a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f7629126640 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad93050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f7629927640 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad90720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.242+0000 7f762a128640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 0x7f76100a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 -- 172.21.15.73:0/3118300271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 msgr2=0x55919ad93050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad93050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 -- 172.21.15.73:0/3118300271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 msgr2=0x55919ad90720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad90720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f7616ffd640 1 -- 172.21.15.73:0/943682567 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f76200f0d60 con 0x7f76241065a0 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 -- 172.21.15.73:0/3118300271 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55919aac97d0 con 0x55919ad87760 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f762a128640 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f761c0139d0 tx=0x7f761c013e90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf415bd33ae49702 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f760ffff640 1 -- 172.21.15.73:0/3118300271 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c013050 con 0x55919ad87760 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f760ffff640 1 -- 172.21.15.73:0/3118300271 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761c022070 con 0x55919ad87760 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.243+0000 7f760ffff640 1 -- 172.21.15.73:0/3118300271 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761c0126b0 con 0x55919ad87760 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- 172.21.15.73:0/3118300271 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 msgr2=0x55919ad87b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762a128640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 0x7f76100a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f761c021000 tx=0x7f761c01f000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 crypto rx=0x7f761c0139d0 tx=0x7f761c013e90 comp rx=0 tx=0).stop 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- 172.21.15.73:0/3118300271 shutdown_connections 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad93050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad90720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 --2- 172.21.15.73:0/3118300271 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad87b40 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- 172.21.15.73:0/3118300271 >> 172.21.15.73:0/3118300271 conn(0x55919ad81e90 msgr2=0x55919ad82270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- 172.21.15.73:0/3118300271 shutdown_connections 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- 172.21.15.73:0/3118300271 wait complete. 2024-03-09T00:11:30.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 -- start start 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.244+0000 7f762e768740 1 Processor -- start 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 0x55919ad90340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad8dc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919ad93a50 con 0x55919ad88080 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919aca70a0 con 0x55919ad90c60 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55919ad93bd0 con 0x55919ad87760 2024-03-09T00:11:30.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 10 client.0 _ll_register_callbacks cb 0x55919aca6a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39148/0 (socket says 172.21.15.73:39148) 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 -- 172.21.15.73:0/3084178431 learned_addr learned my addr 172.21.15.73:0/3084178431 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629927640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 0x55919ad90340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762a128640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad8dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 -- 172.21.15.73:0/3084178431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 msgr2=0x55919ad90340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 0x55919ad90340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 -- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 msgr2=0x55919ad8dc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad8dc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 -- 172.21.15.73:0/3084178431 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55919a9ff3f0 con 0x55919ad88080 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629927640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 0x55919ad90340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f7629126640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7620006010 tx=0x7f76200bcfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c404f887b4a08739 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f760d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f762004b040 con 0x55919ad88080 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f76200bd5a0 con 0x55919ad88080 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.245+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f761c022070 con 0x55919ad88080 2024-03-09T00:11:30.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76200bd140 con 0x55919ad88080 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 10 client.16425 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55919a9bf0f0 con 0x55919ad88080 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761c022070 con 0x55919ad88080 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f76200c5450 con 0x55919ad88080 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 10 client.16425 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 10 client.16425 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 20 client.16425 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919ada05c0 0x55919ada29d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 4 client.16425 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55919acf8340 con 0x55919ada05c0 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.246+0000 7f7629927640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919ada05c0 0x55919ada29d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f7629927640 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919ada05c0 0x55919ada29d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4fbe395ee4598adb server_cookie=1a379ef20a6a247f in_seq=0 out_seq=0 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f760d7fa640 10 client.16425 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f760d7fa640 1 -- 172.21.15.73:0/3084178431 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55919acf8340 con 0x55919ada05c0 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f760d7fa640 10 client.16425 handle_command_reply: tid=0 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 client.16425 shutdown 2024-03-09T00:11:30.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 msgr2=0x55919ad8d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 secure :-1 s=READY pgs=757 cs=0 l=1 rev1=1 crypto rx=0x7f7620006010 tx=0x7f76200bcfd0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 shutdown_connections 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919ada05c0 0x55919ada29d0 crc :-1 s=READY pgs=121 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad8dc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad88080 0x55919ad8d6d0 unknown :-1 s=CLOSED pgs=757 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.247+0000 7f760d7fa640 0 client.16425 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.248+0000 7f762e768740 1 --2- 172.21.15.73:0/3084178431 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad87760 0x55919ad90340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.248+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 >> 172.21.15.73:0/3084178431 conn(0x55919ad81e90 msgr2=0x55919ad83be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.248+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 shutdown_connections 2024-03-09T00:11:30.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.248+0000 7f762e768740 1 -- 172.21.15.73:0/3084178431 wait complete. 2024-03-09T00:11:30.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.248+0000 7f762e768740 20 client.16425 trim_cache size 0 max 16384 2024-03-09T00:11:30.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.262+0000 7f762e768740 1 Processor -- start 2024-03-09T00:11:30.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 -- start start 2024-03-09T00:11:30.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad88080 0x55919ad90340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad97e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241ac5c0 con 0x55919ad87760 2024-03-09T00:11:30.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241aeb80 con 0x55919ad90c60 2024-03-09T00:11:30.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f762e768740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241acbc0 con 0x55919ad88080 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39162/0 (socket says 172.21.15.73:39162) 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 -- 172.21.15.73:0/1094419858 learned_addr learned my addr 172.21.15.73:0/1094419858 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 -- 172.21.15.73:0/1094419858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad88080 msgr2=0x55919ad90340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad88080 0x55919ad90340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 -- 172.21.15.73:0/1094419858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 msgr2=0x55919ad97e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad97e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 -- 172.21.15.73:0/1094419858 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55919ad8bc40 con 0x55919ad87760 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f7629927640 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55919ada32f0 tx=0x7f7618005c90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58c9cdea6eb5e6f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f760f7fe640 1 -- 172.21.15.73:0/1094419858 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180040f0 con 0x55919ad87760 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f760f7fe640 1 -- 172.21.15.73:0/1094419858 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761801b8b0 con 0x55919ad87760 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.263+0000 7f760f7fe640 1 -- 172.21.15.73:0/1094419858 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76180047a0 con 0x55919ad87760 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- 172.21.15.73:0/1094419858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 msgr2=0x55919ad8fe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 crypto rx=0x55919ada32f0 tx=0x7f7618005c90 comp rx=0 tx=0).stop 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- 172.21.15.73:0/1094419858 shutdown_connections 2024-03-09T00:11:30.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad90c60 0x55919ad97e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad88080 0x55919ad90340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 --2- 172.21.15.73:0/1094419858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8fe00 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- 172.21.15.73:0/1094419858 >> 172.21.15.73:0/1094419858 conn(0x55919ad81e90 msgr2=0x55919ad8d690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- 172.21.15.73:0/1094419858 shutdown_connections 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- 172.21.15.73:0/1094419858 wait complete. 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 -- start start 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.264+0000 7f762e768740 1 Processor -- start 2024-03-09T00:11:30.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad8a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad8ad30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241ad200 con 0x55919ad87760 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241ae520 con 0x55919ad88080 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76241ad860 con 0x55919ad90c60 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 10 client.0 _ll_register_callbacks cb 0x7f7620009580 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39172/0 (socket says 172.21.15.73:39172) 2024-03-09T00:11:30.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 -- 172.21.15.73:0/4004368295 learned_addr learned my addr 172.21.15.73:0/4004368295 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629126640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad8a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 -- 172.21.15.73:0/4004368295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 msgr2=0x55919ad8ad30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad8ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 -- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 msgr2=0x55919ad8a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad8a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 -- 172.21.15.73:0/4004368295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55919adf6af0 con 0x55919ad87760 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629126640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad8a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f7629927640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f761801ba60 tx=0x7f7618005250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c002c066e94ce39 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f760d7fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f761802c030 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f761801b3d0 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.265+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f761801b8b0 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7618036050 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 10 client.16431 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55919adf7d20 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f761801b8b0 con 0x55919ad87760 2024-03-09T00:11:30.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7618024040 con 0x55919ad87760 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 10 client.16431 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 10 client.16431 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 20 client.16431 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919adf8410 0x55919adfa830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 4 client.16431 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55919acf2690 con 0x55919adf8410 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.266+0000 7f7629126640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919adf8410 0x55919adfa830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f760d7fa640 10 client.16431 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f7629126640 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919adf8410 0x55919adfa830 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc1974dde64616d server_cookie=7b197fad21b6772b in_seq=0 out_seq=0 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f760d7fa640 1 -- 172.21.15.73:0/4004368295 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55919acf2690 con 0x55919adf8410 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f760d7fa640 10 client.16431 handle_command_reply: tid=0 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 client.16431 shutdown 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 msgr2=0x55919ad8a2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 crypto rx=0x7f761801ba60 tx=0x7f7618005250 comp rx=0 tx=0).stop 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 shutdown_connections 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55919adf8410 0x55919adfa830 crc :-1 s=READY pgs=122 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55919ad90c60 0x55919ad8ad30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55919ad88080 0x55919ad8a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f760d7fa640 0 client.16431 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 --2- 172.21.15.73:0/4004368295 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55919ad87760 0x55919ad8a2b0 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 >> 172.21.15.73:0/4004368295 conn(0x55919ad81e90 msgr2=0x55919ad8bf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 shutdown_connections 2024-03-09T00:11:30.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 1 -- 172.21.15.73:0/4004368295 wait complete. 2024-03-09T00:11:30.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.267+0000 7f762e768740 20 client.16431 trim_cache size 0 max 16384 2024-03-09T00:11:30.275 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:11:30.276 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:11:30.276 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:11:30.276 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:11:30.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 msgr2=0x7f76100a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 0x7f76100a4eb0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f761c021000 tx=0x7f761c01f000 comp rx=0 tx=0).stop 2024-03-09T00:11:30.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 msgr2=0x7f762419dc20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 crypto rx=0x7f7620000c00 tx=0x7f762002cc70 comp rx=0 tx=0).stop 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 shutdown_connections 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7624197890 0x7f7624197d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76100a29f0 0x7f76100a4eb0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f76241065a0 0x7f762419dc20 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 --2- 172.21.15.73:0/943682567 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7624068740 0x7f762419d6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 >> 172.21.15.73:0/943682567 conn(0x7f76240772f0 msgr2=0x7f76240fe100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 shutdown_connections 2024-03-09T00:11:30.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.268+0000 7f762bb8b640 1 -- 172.21.15.73:0/943682567 wait complete. 2024-03-09T00:11:30.279 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:11:30.279 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:11:30.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.384+0000 7fcc10cf8640 1 -- 172.21.15.73:0/3922483728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 msgr2=0x7fcc0c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.384+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/3922483728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c10e1a0 secure :-1 s=READY pgs=760 cs=0 l=1 rev1=1 crypto rx=0x7fcbfc009900 tx=0x7fcbfc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.384+0000 7fcc10cf8640 1 -- 172.21.15.73:0/3922483728 shutdown_connections 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/3922483728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c10e1a0 unknown :-1 s=CLOSED pgs=760 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/3922483728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101eb0 0x7fcc0c105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/3922483728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c101500 0x7fcc0c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 -- 172.21.15.73:0/3922483728 >> 172.21.15.73:0/3922483728 conn(0x7fcc0c0fcd80 msgr2=0x7fcc0c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 -- 172.21.15.73:0/3922483728 shutdown_connections 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 -- 172.21.15.73:0/3922483728 wait complete. 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 Processor -- start 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 -- start start 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 0x7fcc0c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.385+0000 7fcc10cf8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 0x7fcc0c197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc10cf8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c1116e0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc10cf8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c111560 con 0x7fcc0c101500 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc10cf8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc0c111860 con 0x7fcc0c197a20 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39186/0 (socket says 172.21.15.73:39186) 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 -- 172.21.15.73:0/359788879 learned_addr learned my addr 172.21.15.73:0/359788879 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc0a59c640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 0x7fcc0c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 -- 172.21.15.73:0/359788879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 msgr2=0x7fcc0c197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc0ad9d640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 0x7fcc0c197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 0x7fcc0c197ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 -- 172.21.15.73:0/359788879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 msgr2=0x7fcc0c19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 0x7fcc0c19d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 -- 172.21.15.73:0/359788879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0c198540 con 0x7fcc0c1065d0 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc0a59c640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 0x7fcc0c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc0ad9d640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 0x7fcc0c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc09d9b640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000ccb0 tx=0x7fcc0000a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc00004b20 con 0x7fcc0c1065d0 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc00004cc0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc0c198830 con 0x7fcc0c1065d0 2024-03-09T00:11:30.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc0000f830 con 0x7fcc0c1065d0 2024-03-09T00:11:30.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.386+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc0c1a4a30 con 0x7fcc0c1065d0 2024-03-09T00:11:30.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.387+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc0c0680e0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.388+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcc0000f9d0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.389+0000 7fcbfb7fe640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 0x7fcbf40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.389+0000 7fcc0a59c640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 0x7fcbf40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.389+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcc000cf6d0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.389+0000 7fcc0a59c640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 0x7fcbf40a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0fd220 tx=0x7fcc04008040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.393+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcc00014030 con 0x7fcc0c1065d0 2024-03-09T00:11:30.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:30 smithi073 ceph-mon[32309]: pgmap v142: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:30.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:30 smithi102 ceph-mon[36106]: pgmap v142: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:30.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:30 smithi179 ceph-mon[36637]: pgmap v142: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:30.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.604+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcc0c101eb0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.605+0000 7fcbfb7fe640 1 -- 172.21.15.73:0/359788879 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fcc0009c0a0 con 0x7fcc0c1065d0 2024-03-09T00:11:30.601 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:11:30.602 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:11:30.602 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:11:30.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 msgr2=0x7fcbf40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 0x7fcbf40a4e30 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcc0c0fd220 tx=0x7fcc04008040 comp rx=0 tx=0).stop 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 msgr2=0x7fcc0c19dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 crypto rx=0x7fcc0000ccb0 tx=0x7fcc0000a620 comp rx=0 tx=0).stop 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 shutdown_connections 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcc0c197a20 0x7fcc0c197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcbf40a2970 0x7fcbf40a4e30 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcc0c1065d0 0x7fcc0c19dd60 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 --2- 172.21.15.73:0/359788879 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcc0c101500 0x7fcc0c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 >> 172.21.15.73:0/359788879 conn(0x7fcc0c0fcd80 msgr2=0x7fcc0c0ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 shutdown_connections 2024-03-09T00:11:30.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.607+0000 7fcc10cf8640 1 -- 172.21.15.73:0/359788879 wait complete. 2024-03-09T00:11:30.614 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:11:30.615 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:11:30.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.721+0000 7f69c0b0a640 1 -- 172.21.15.73:0/365127039 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc100d90 msgr2=0x7f69bc104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.721+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/365127039 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc100d90 0x7f69bc104dd0 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 crypto rx=0x7f69ac009900 tx=0x7f69ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:11:30.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.721+0000 7f69c0b0a640 1 -- 172.21.15.73:0/365127039 shutdown_connections 2024-03-09T00:11:30.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.721+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/365127039 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.721+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/365127039 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc100d90 0x7f69bc104dd0 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/365127039 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69bc1003e0 0x7f69bc1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- 172.21.15.73:0/365127039 >> 172.21.15.73:0/365127039 conn(0x7f69bc0773a0 msgr2=0x7f69bc0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- 172.21.15.73:0/365127039 shutdown_connections 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- 172.21.15.73:0/365127039 wait complete. 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 Processor -- start 2024-03-09T00:11:30.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- start start 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc196cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69bc197210 0x7f69bc1a5e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc110540 con 0x7f69bc1003e0 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1103c0 con 0x7f69bc1054b0 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.722+0000 7f69c0b0a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1106c0 con 0x7f69bc197210 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39202/0 (socket says 172.21.15.73:39202) 2024-03-09T00:11:30.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 -- 172.21.15.73:0/2656902819 learned_addr learned my addr 172.21.15.73:0/2656902819 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc196cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 -- 172.21.15.73:0/2656902819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69bc197210 msgr2=0x7f69bc1a5e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69bc197210 0x7f69bc1a5e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 -- 172.21.15.73:0/2656902819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 msgr2=0x7f69bc196cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc196cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 -- 172.21.15.73:0/2656902819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69bc1a63d0 con 0x7f69bc1003e0 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ba59c640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c9a0 tx=0x7f69b000ce60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ab7fe640 1 -- 172.21.15.73:0/2656902819 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b0007b50 con 0x7f69bc1003e0 2024-03-09T00:11:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ab7fe640 1 -- 172.21.15.73:0/2656902819 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b0005500 con 0x7f69bc1003e0 2024-03-09T00:11:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69ab7fe640 1 -- 172.21.15.73:0/2656902819 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b000f450 con 0x7f69bc1003e0 2024-03-09T00:11:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69bc1a66c0 con 0x7f69bc1003e0 2024-03-09T00:11:30.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.723+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69bc1a6b90 con 0x7f69bc1003e0 2024-03-09T00:11:30.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69ab7fe640 1 -- 172.21.15.73:0/2656902819 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f69b0005000 con 0x7f69bc1003e0 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 Processor -- start 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 -- start start 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d39ac0 0x5571d8d42160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d44a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.726+0000 7f69c37bf740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8d00360 con 0x5571d8d391a0 2024-03-09T00:11:30.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69c37bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8c9cec0 con 0x5571d8d39ac0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69c37bf740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8c45d30 con 0x5571d8d426a0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69ab7fe640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 0x7f69a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39218/0 (socket says 172.21.15.73:39218) 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 -- 172.21.15.73:0/3793651667 learned_addr learned my addr 172.21.15.73:0/3793651667 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69ba59c640 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d39ac0 0x5571d8d42160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 0x7f69a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d44a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69ab7fe640 1 -- 172.21.15.73:0/2656902819 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f69b00cf090 con 0x7f69bc1003e0 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 -- 172.21.15.73:0/3793651667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 msgr2=0x5571d8d44a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d44a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 -- 172.21.15.73:0/3793651667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d39ac0 msgr2=0x5571d8d42160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d39ac0 0x5571d8d42160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 -- 172.21.15.73:0/3793651667 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571d8c44bc0 con 0x5571d8d391a0 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f69b4013b90 tx=0x7f69b4010740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dd06ff14789508b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69bad9d640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 0x7f69a40a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69b4010c40 tx=0x7f69b4019000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69a8ff9640 1 -- 172.21.15.73:0/3793651667 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4013620 con 0x5571d8d391a0 2024-03-09T00:11:30.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69a8ff9640 1 -- 172.21.15.73:0/3793651667 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b4021070 con 0x5571d8d391a0 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.727+0000 7f69a8ff9640 1 -- 172.21.15.73:0/3793651667 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69b4004230 con 0x5571d8d391a0 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 -- 172.21.15.73:0/3793651667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 msgr2=0x5571d8d39580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 crypto rx=0x7f69b4013b90 tx=0x7f69b4010740 comp rx=0 tx=0).stop 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 -- 172.21.15.73:0/3793651667 shutdown_connections 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d44a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d39ac0 0x5571d8d42160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3793651667 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d391a0 0x5571d8d39580 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 -- 172.21.15.73:0/3793651667 >> 172.21.15.73:0/3793651667 conn(0x5571d8d338d0 msgr2=0x5571d8d33cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 -- 172.21.15.73:0/3793651667 shutdown_connections 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.728+0000 7f69c37bf740 1 -- 172.21.15.73:0/3793651667 wait complete. 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 -- start start 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 Processor -- start 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d3f730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d401b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8c3b4a0 con 0x5571d8d39ac0 2024-03-09T00:11:30.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8c98f60 con 0x5571d8d391a0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5571d8d45430 con 0x5571d8d426a0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69c37bf740 10 client.0 _ll_register_callbacks cb 0x5571d8c96d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39232/0 (socket says 172.21.15.73:39232) 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 -- 172.21.15.73:0/149298718 learned_addr learned my addr 172.21.15.73:0/149298718 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69ba59c640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d3f730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69bad9d640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 -- 172.21.15.73:0/149298718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 msgr2=0x5571d8d401b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 -- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 msgr2=0x5571d8d3f730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d3f730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 -- 172.21.15.73:0/149298718 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571d89b13f0 con 0x5571d8d39ac0 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f69bc06aa60 tx=0x7f69ac02ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1bce64c327398930 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.729+0000 7f69a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac043070 con 0x5571d8d39ac0 2024-03-09T00:11:30.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f69ac0048c0 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69b4021070 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69ac004ac0 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 10 client.16449 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5571d8a7b7d0 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69b4021070 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f69ac0037b0 con 0x5571d8d39ac0 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 10 client.16449 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:30.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 10 client.16449 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 20 client.16449 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8d528a0 0x5571d8d54c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 4 client.16449 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5571d8caa400 con 0x5571d8d528a0 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.730+0000 7f69ba59c640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8d528a0 0x5571d8d54c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69ba59c640 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8d528a0 0x5571d8d54c90 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e19dae19e7af196e server_cookie=2f7f0373397b8c42 in_seq=0 out_seq=0 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69a1ffb640 10 client.16449 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69a1ffb640 1 -- 172.21.15.73:0/149298718 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7f69b007fa90 con 0x5571d8d528a0 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69a1ffb640 10 client.16449 handle_command_reply: tid=0 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69c37bf740 1 client.16449 shutdown 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 msgr2=0x5571d8d3fc70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.731+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 crypto rx=0x7f69bc06aa60 tx=0x7f69ac02ce30 comp rx=0 tx=0).stop 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 shutdown_connections 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8d528a0 0x5571d8d54c90 crc :-1 s=READY pgs=123 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d401b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3fc70 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 --2- 172.21.15.73:0/149298718 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d3f730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69a1ffb640 0 client.16449 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 >> 172.21.15.73:0/149298718 conn(0x5571d8d338d0 msgr2=0x5571d8d35620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 shutdown_connections 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 1 -- 172.21.15.73:0/149298718 wait complete. 2024-03-09T00:11:30.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.732+0000 7f69c37bf740 20 client.16449 trim_cache size 0 max 16384 2024-03-09T00:11:30.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 Processor -- start 2024-03-09T00:11:30.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 -- start start 2024-03-09T00:11:30.747 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d41e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d3e540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1ae8f0 con 0x5571d8d39ac0 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1b0eb0 con 0x5571d8d391a0 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69c37bf740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1aeef0 con 0x5571d8d426a0 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39234/0 (socket says 172.21.15.73:39234) 2024-03-09T00:11:30.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.743+0000 7f69b9d9b640 1 -- 172.21.15.73:0/2972253691 learned_addr learned my addr 172.21.15.73:0/2972253691 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 -- 172.21.15.73:0/2972253691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 msgr2=0x5571d8d3e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d3e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 -- 172.21.15.73:0/2972253691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 msgr2=0x5571d8d41e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d41e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 -- 172.21.15.73:0/2972253691 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5571d8d398f0 con 0x5571d8d39ac0 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f69ac000c00 tx=0x7f69ac005920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4dbc445bcf682cee server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69a3fff640 1 -- 172.21.15.73:0/2972253691 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac043030 con 0x5571d8d39ac0 2024-03-09T00:11:30.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69a3fff640 1 -- 172.21.15.73:0/2972253691 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69ac004c60 con 0x5571d8d39ac0 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69a3fff640 1 -- 172.21.15.73:0/2972253691 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac005170 con 0x5571d8d39ac0 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 -- 172.21.15.73:0/2972253691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 msgr2=0x5571d8d3e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 secure :-1 s=READY pgs=766 cs=0 l=1 rev1=1 crypto rx=0x7f69ac000c00 tx=0x7f69ac005920 comp rx=0 tx=0).stop 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 -- 172.21.15.73:0/2972253691 shutdown_connections 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8d3e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d3e000 unknown :-1 s=CLOSED pgs=766 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 --2- 172.21.15.73:0/2972253691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d41e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.744+0000 7f69c37bf740 1 -- 172.21.15.73:0/2972253691 >> 172.21.15.73:0/2972253691 conn(0x5571d8d338d0 msgr2=0x5571d8d3a9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- 172.21.15.73:0/2972253691 shutdown_connections 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- 172.21.15.73:0/2972253691 wait complete. 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- start start 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 Processor -- start 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d48440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8da8da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1af530 con 0x5571d8d39ac0 2024-03-09T00:11:30.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1b0850 con 0x5571d8d391a0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69bc1afb90 con 0x5571d8d426a0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69c37bf740 10 client.0 _ll_register_callbacks cb 0x7f69b4007460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39240/0 (socket says 172.21.15.73:39240) 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 -- 172.21.15.73:0/3974315118 learned_addr learned my addr 172.21.15.73:0/3974315118 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69bad9d640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8da8da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69ba59c640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d48440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 -- 172.21.15.73:0/3974315118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 msgr2=0x5571d8da8da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8da8da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 -- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 msgr2=0x5571d8d48440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d48440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69b9d9b640 1 -- 172.21.15.73:0/3974315118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5571d8da92e0 con 0x5571d8d39ac0 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.745+0000 7f69bad9d640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8da8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69b9d9b640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f69ac003af0 tx=0x7f69ac02c740 comp rx=0 tx=0).ready entity=mon.0 client_cookie=86ce15a26cd4e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69ac04e040 con 0x5571d8d39ac0 2024-03-09T00:11:30.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f69ac005110 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69ac004c60 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f69ac063050 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 10 client.16455 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5571d8daa4c0 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69ac004c60 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f69ac048bc0 con 0x5571d8d39ac0 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 10 client.16455 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:30.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 10 client.16455 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 20 client.16455 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8daab30 0x5571d8dacf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 4 client.16455 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5571d8ca4750 con 0x5571d8daab30 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.746+0000 7f69ba59c640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8daab30 0x5571d8dacf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69a1ffb640 10 client.16455 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69ba59c640 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8daab30 0x5571d8dacf50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d1acfb0a207a1674 server_cookie=f4cb479dc86c5ae8 in_seq=0 out_seq=0 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69a1ffb640 1 -- 172.21.15.73:0/3974315118 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5571d8ca4750 con 0x5571d8daab30 2024-03-09T00:11:30.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69a1ffb640 10 client.16455 handle_command_reply: tid=0 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 client.16455 shutdown 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 msgr2=0x5571d8d48980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 crypto rx=0x7f69ac003af0 tx=0x7f69ac02c740 comp rx=0 tx=0).stop 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 shutdown_connections 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5571d8daab30 0x5571d8dacf50 crc :-1 s=READY pgs=124 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5571d8d426a0 0x5571d8da8da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5571d8d39ac0 0x5571d8d48980 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 --2- 172.21.15.73:0/3974315118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5571d8d391a0 0x5571d8d48440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69a1ffb640 0 client.16455 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 >> 172.21.15.73:0/3974315118 conn(0x5571d8d338d0 msgr2=0x5571d8d3df40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 shutdown_connections 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 1 -- 172.21.15.73:0/3974315118 wait complete. 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.747+0000 7f69c37bf740 20 client.16455 trim_cache size 0 max 16384 2024-03-09T00:11:30.757 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:11:30.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 msgr2=0x7f69a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 0x7f69a40a4e30 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f69b4010c40 tx=0x7f69b4019000 comp rx=0 tx=0).stop 2024-03-09T00:11:30.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 msgr2=0x7f69bc196790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:30.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 crypto rx=0x7f69b000c9a0 tx=0x7f69b000ce60 comp rx=0 tx=0).stop 2024-03-09T00:11:30.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 shutdown_connections 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69bc197210 0x7f69bc1a5e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69bc1054b0 0x7f69bc196cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69a40a2970 0x7f69a40a4e30 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 --2- 172.21.15.73:0/2656902819 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69bc1003e0 0x7f69bc196790 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 >> 172.21.15.73:0/2656902819 conn(0x7f69bc0773a0 msgr2=0x7f69bc0fe2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 shutdown_connections 2024-03-09T00:11:30.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:30.748+0000 7f69c0b0a640 1 -- 172.21.15.73:0/2656902819 wait complete. 2024-03-09T00:11:31.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:31 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/359788879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:11:31.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:31 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/359788879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:11:31.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:31 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/359788879' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:11:31.761 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:11:31.762 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:11:31.762 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 -- 172.21.15.73:0/2530630972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc100d80 msgr2=0x7ff3bc104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/2530630972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc100d80 0x7ff3bc104dc0 secure :-1 s=READY pgs=768 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4009900 tx=0x7ff3a402bc30 comp rx=0 tx=0).stop 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 -- 172.21.15.73:0/2530630972 shutdown_connections 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/2530630972 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/2530630972 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc100d80 0x7ff3bc104dc0 unknown :-1 s=CLOSED pgs=768 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/2530630972 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 0x7ff3bc1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 -- 172.21.15.73:0/2530630972 >> 172.21.15.73:0/2530630972 conn(0x7ff3bc0772f0 msgr2=0x7ff3bc077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 -- 172.21.15.73:0/2530630972 shutdown_connections 2024-03-09T00:11:31.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.883+0000 7ff3c11fa640 1 -- 172.21.15.73:0/2530630972 wait complete. 2024-03-09T00:11:31.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 Processor -- start 2024-03-09T00:11:31.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 -- start start 2024-03-09T00:11:31.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 0x7ff3bc198ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 0x7ff3bc1a3b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc110610 con 0x7ff3bc1054a0 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc110490 con 0x7ff3bc1995a0 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3c11fa640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc110790 con 0x7ff3bc1003d0 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.884+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39256/0 (socket says 172.21.15.73:39256) 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 -- 172.21.15.73:0/3576653638 learned_addr learned my addr 172.21.15.73:0/3576653638 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 0x7ff3bc198ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 0x7ff3bc1a3b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 -- 172.21.15.73:0/3576653638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 msgr2=0x7ff3bc198ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 0x7ff3bc198ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 -- 172.21.15.73:0/3576653638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 msgr2=0x7ff3bc1a3b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 0x7ff3bc1a3b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 -- 172.21.15.73:0/3576653638 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc1a4320 con 0x7ff3bc1054a0 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 0x7ff3bc1a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b259c640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4000c00 tx=0x7ff3a402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b8d99640 1 -- 172.21.15.73:0/3576653638 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4041070 con 0x7ff3bc1054a0 2024-03-09T00:11:31.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b8d99640 1 -- 172.21.15.73:0/3576653638 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3a4004450 con 0x7ff3bc1054a0 2024-03-09T00:11:31.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc1a45b0 con 0x7ff3bc1054a0 2024-03-09T00:11:31.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3b8d99640 1 -- 172.21.15.73:0/3576653638 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a4003e50 con 0x7ff3bc1054a0 2024-03-09T00:11:31.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.885+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc1a4aa0 con 0x7ff3bc1054a0 2024-03-09T00:11:31.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.888+0000 7ff3b8d99640 1 -- 172.21.15.73:0/3576653638 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff3a4044070 con 0x7ff3bc1054a0 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.888+0000 7ff3c3dd6740 1 Processor -- start 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.888+0000 7ff3c3dd6740 1 -- start start 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507c6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56325072ab40 con 0x5632507d4690 2024-03-09T00:11:31.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632506d3d80 con 0x5632507c6a30 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563250726be0 con 0x5632507c7350 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b8d99640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 0x7ff3a80a4fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39258/0 (socket says 172.21.15.73:39258) 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 -- 172.21.15.73:0/599212611 learned_addr learned my addr 172.21.15.73:0/599212611 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 0x7ff3a80a4fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507c6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b8d99640 1 -- 172.21.15.73:0/3576653638 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff3a40f0e60 con 0x7ff3bc1054a0 2024-03-09T00:11:31.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 -- 172.21.15.73:0/599212611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 msgr2=0x5632507d4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 -- 172.21.15.73:0/599212611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 msgr2=0x5632507c6e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507c6e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 -- 172.21.15.73:0/599212611 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632505077e0 con 0x5632507d4690 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 0x7ff3a80a4fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009440 tx=0x7ff3b4009e20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b259c640 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40f11c0 tx=0x7ff3a403c000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eaad987b2b38c769 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.889+0000 7ff3b159a640 1 -- 172.21.15.73:0/599212611 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40bcba0 con 0x5632507d4690 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3b159a640 1 -- 172.21.15.73:0/599212611 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3a40ee0b0 con 0x5632507d4690 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3b159a640 1 -- 172.21.15.73:0/599212611 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a40ee4a0 con 0x5632507d4690 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/599212611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 msgr2=0x5632507d6a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40f11c0 tx=0x7ff3a403c000 comp rx=0 tx=0).stop 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/599212611 shutdown_connections 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d6a80 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/599212611 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507c6e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/599212611 >> 172.21.15.73:0/599212611 conn(0x5632507c14b0 msgr2=0x5632507c1890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/599212611 shutdown_connections 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.890+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/599212611 wait complete. 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 -- start start 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 Processor -- start 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d1490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507d4690 0x5632507d1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632507d7290 con 0x5632507c6a30 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632506e71c0 con 0x5632507d4690 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632507d7410 con 0x5632507c7350 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3c3dd6740 10 client.0 _ll_register_callbacks cb 0x5632506d67a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39266/0 (socket says 172.21.15.73:39266) 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 -- 172.21.15.73:0/949035070 learned_addr learned my addr 172.21.15.73:0/949035070 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 -- 172.21.15.73:0/949035070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 msgr2=0x5632507d1490 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-09T00:11:31.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d1490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 -- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507d4690 msgr2=0x5632507d1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507d4690 0x5632507d1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 -- 172.21.15.73:0/949035070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563250567f70 con 0x5632507c6a30 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009f50 tx=0x7ff3ac0116f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ecd8431520e4e87 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.891+0000 7ff3a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac019070 con 0x5632507c6a30 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff3ac015870 con 0x5632507c6a30 2024-03-09T00:11:31.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x563250700960 con 0x5632507c6a30 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3ac025050 con 0x5632507c6a30 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 10 client.16467 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5632503ff170 con 0x5632507c6a30 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x563250700960 con 0x5632507c6a30 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff3ac014b50 con 0x5632507c6a30 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 10 client.16467 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 10 client.16467 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:31.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 20 client.16467 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5632507e4540 0x5632507e6990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 4 client.16467 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632506ec080 con 0x5632507e4540 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.892+0000 7ff3b259c640 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5632507e4540 0x5632507e6990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.893+0000 7ff3a27fc640 10 client.16467 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.893+0000 7ff3b259c640 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5632507e4540 0x5632507e6990 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4cb0e602b100e63 server_cookie=4626d45925e43b48 in_seq=0 out_seq=0 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.893+0000 7ff3a27fc640 1 -- 172.21.15.73:0/949035070 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5632506ec080 con 0x5632507e4540 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.893+0000 7ff3a27fc640 10 client.16467 handle_command_reply: tid=0 2024-03-09T00:11:31.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.893+0000 7ff3c3dd6740 1 client.16467 shutdown 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 msgr2=0x5632507d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 secure :-1 s=READY pgs=771 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009f50 tx=0x7ff3ac0116f0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 shutdown_connections 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5632507e4540 0x5632507e6990 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507d4690 0x5632507d1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d1490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3a27fc640 0 client.16467 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:31.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/949035070 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c6a30 0x5632507d4100 unknown :-1 s=CLOSED pgs=771 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 >> 172.21.15.73:0/949035070 conn(0x5632507c14b0 msgr2=0x5632507c3200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 shutdown_connections 2024-03-09T00:11:31.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/949035070 wait complete. 2024-03-09T00:11:31.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.894+0000 7ff3c3dd6740 20 client.16467 trim_cache size 0 max 16384 2024-03-09T00:11:31.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 Processor -- start 2024-03-09T00:11:31.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 -- start start 2024-03-09T00:11:31.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507d0490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d3bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1aa520 con 0x5632507d4690 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1acae0 con 0x5632507c6a30 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.909+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1aab20 con 0x5632507c7350 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39280/0 (socket says 172.21.15.73:39280) 2024-03-09T00:11:31.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 -- 172.21.15.73:0/1186481833 learned_addr learned my addr 172.21.15.73:0/1186481833 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507d0490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3b259c640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d3bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 -- 172.21.15.73:0/1186481833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 msgr2=0x5632507d3bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d3bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 -- 172.21.15.73:0/1186481833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 msgr2=0x5632507d0490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507d0490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 -- 172.21.15.73:0/1186481833 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632507d1f40 con 0x5632507d4690 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:11:31.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3b259c640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400a740 tx=0x7ff3b4019000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7497194705c8b36 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3b0d99640 1 -- 172.21.15.73:0/1186481833 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4028070 con 0x5632507d4690 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3b0d99640 1 -- 172.21.15.73:0/1186481833 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b400ac70 con 0x5632507d4690 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.910+0000 7ff3b0d99640 1 -- 172.21.15.73:0/1186481833 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3b4013040 con 0x5632507d4690 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/1186481833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 msgr2=0x5632507d4100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 secure :-1 s=READY pgs=772 cs=0 l=1 rev1=1 crypto rx=0x7ff3b400a740 tx=0x7ff3b4019000 comp rx=0 tx=0).stop 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/1186481833 shutdown_connections 2024-03-09T00:11:31.914 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507d4690 0x5632507d4100 unknown :-1 s=CLOSED pgs=772 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507c7350 0x5632507d3bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/1186481833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507d0490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/1186481833 >> 172.21.15.73:0/1186481833 conn(0x5632507c14b0 msgr2=0x5632507cfd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/1186481833 shutdown_connections 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/1186481833 wait complete. 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 -- start start 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 Processor -- start 2024-03-09T00:11:31.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.911+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507da1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507d4690 0x5632507dac50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1ab160 con 0x5632507c7350 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1ac480 con 0x5632507c6a30 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc1ab7c0 con 0x5632507d4690 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 10 client.0 _ll_register_callbacks cb 0x7ff3a4005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39282/0 (socket says 172.21.15.73:39282) 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 -- 172.21.15.73:0/2558314382 learned_addr learned my addr 172.21.15.73:0/2558314382 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3bb59e640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507d4690 0x5632507dac50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 -- 172.21.15.73:0/2558314382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507d4690 msgr2=0x5632507dac50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507da1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507d4690 0x5632507dac50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 -- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 msgr2=0x5632507da1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507da1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 -- 172.21.15.73:0/2558314382 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56325083ac80 con 0x5632507c7350 2024-03-09T00:11:31.917 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3b259c640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40096c0 tx=0x7ff3a40bce40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6d27fc8503ff64d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3a27fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3a404b040 con 0x5632507c7350 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7ff3a40035f0 con 0x5632507c7350 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff3b400ac70 con 0x5632507c7350 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff3a40c1390 con 0x5632507c7350 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 10 client.16473 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.912+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56325083bf10 con 0x5632507c7350 2024-03-09T00:11:31.918 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3b400ac70 con 0x5632507c7350 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff3a40b98b0 con 0x5632507c7350 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 10 client.16473 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 10 client.16473 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 20 client.16473 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56325083c470 0x56325083e890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 4 client.16473 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563250738080 con 0x56325083c470 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56325083c470 0x56325083e890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:31.919 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3a27fc640 10 client.16473 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.913+0000 7ff3bad9d640 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56325083c470 0x56325083e890 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b094be9128860b3b server_cookie=6fa5eea32f161ee2 in_seq=0 out_seq=0 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.914+0000 7ff3a27fc640 1 -- 172.21.15.73:0/2558314382 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563250738080 con 0x56325083c470 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.914+0000 7ff3a27fc640 10 client.16473 handle_command_reply: tid=0 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.914+0000 7ff3c3dd6740 1 client.16473 shutdown 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.914+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 msgr2=0x5632507da710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.914+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 secure :-1 s=READY pgs=773 cs=0 l=1 rev1=1 crypto rx=0x7ff3a40096c0 tx=0x7ff3a40bce40 comp rx=0 tx=0).stop 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 shutdown_connections 2024-03-09T00:11:31.920 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56325083c470 0x56325083e890 crc :-1 s=READY pgs=126 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3a27fc640 0 client.16473 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5632507d4690 0x5632507dac50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5632507c7350 0x5632507da710 unknown :-1 s=CLOSED pgs=773 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 --2- 172.21.15.73:0/2558314382 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5632507c6a30 0x5632507da1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 >> 172.21.15.73:0/2558314382 conn(0x5632507c14b0 msgr2=0x5632507c2900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 shutdown_connections 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 1 -- 172.21.15.73:0/2558314382 wait complete. 2024-03-09T00:11:31.921 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c3dd6740 20 client.16473 trim_cache size 0 max 16384 2024-03-09T00:11:31.922 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:11:31.922 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:11:31.922 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "00e05d7d-ed88-4d6e-b3c3-03978523d737", 2024-03-09T00:11:31.922 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:11:31.922 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:11:31.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 msgr2=0x7ff3a80a4fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 0x7ff3a80a4fd0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7ff3b4009440 tx=0x7ff3b4009e20 comp rx=0 tx=0).stop 2024-03-09T00:11:31.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 msgr2=0x7ff3bc199010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:31.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 crypto rx=0x7ff3a4000c00 tx=0x7ff3a402cc70 comp rx=0 tx=0).stop 2024-03-09T00:11:31.924 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 shutdown_connections 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff3bc1995a0 0x7ff3bc1a3b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff3a80a2b10 0x7ff3a80a4fd0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff3bc1054a0 0x7ff3bc199010 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 --2- 172.21.15.73:0/3576653638 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff3bc1003d0 0x7ff3bc198ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 >> 172.21.15.73:0/3576653638 conn(0x7ff3bc0772f0 msgr2=0x7ff3bc0fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 shutdown_connections 2024-03-09T00:11:31.925 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:31.915+0000 7ff3c11fa640 1 -- 172.21.15.73:0/3576653638 wait complete. 2024-03-09T00:11:31.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:11:31.926 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:11:32.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/1464738069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f01054a0 msgr2=0x7fe0f010d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/1464738069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f01054a0 0x7fe0f010d070 secure :-1 s=READY pgs=774 cs=0 l=1 rev1=1 crypto rx=0x7fe0e8009ca0 tx=0x7fe0e802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/1464738069 shutdown_connections 2024-03-09T00:11:32.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/1464738069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f01054a0 0x7fe0f010d070 unknown :-1 s=CLOSED pgs=774 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/1464738069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f0100d80 0x7fe0f0104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/1464738069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 0x7fe0f01007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/1464738069 >> 172.21.15.73:0/1464738069 conn(0x7fe0f00772f0 msgr2=0x7fe0f0077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/1464738069 shutdown_connections 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.038+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/1464738069 wait complete. 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 Processor -- start 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 -- start start 2024-03-09T00:11:32.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 0x7fe0f0072be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f01054a0 0x7fe0f006eba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f006b470 con 0x7fe0f0073120 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f006b2f0 con 0x7fe0f01003d0 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f6a9a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f006b5f0 con 0x7fe0f01054a0 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f5037640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f5037640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39308/0 (socket says 172.21.15.73:39308) 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f5037640 1 -- 172.21.15.73:0/4251161422 learned_addr learned my addr 172.21.15.73:0/4251161422 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0f4836640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 0x7fe0f0072be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.039+0000 7fe0effff640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f01054a0 0x7fe0f006eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 -- 172.21.15.73:0/4251161422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f01054a0 msgr2=0x7fe0f006eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f01054a0 0x7fe0f006eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 -- 172.21.15.73:0/4251161422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 msgr2=0x7fe0f0072be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 0x7fe0f0072be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 -- 172.21.15.73:0/4251161422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0f006f9e0 con 0x7fe0f0073120 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f5037640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe0e802c700 tx=0x7fe0e8032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0edffb640 1 -- 172.21.15.73:0/4251161422 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8041020 con 0x7fe0f0073120 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0edffb640 1 -- 172.21.15.73:0/4251161422 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0e8034cd0 con 0x7fe0f0073120 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0f01acc00 con 0x7fe0f0073120 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0edffb640 1 -- 172.21.15.73:0/4251161422 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e8039040 con 0x7fe0f0073120 2024-03-09T00:11:32.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.040+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0f01ad0f0 con 0x7fe0f0073120 2024-03-09T00:11:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0edffb640 1 -- 172.21.15.73:0/4251161422 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe0e8044020 con 0x7fe0f0073120 2024-03-09T00:11:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 Processor -- start 2024-03-09T00:11:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 -- start start 2024-03-09T00:11:32.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860f6080 0x5604860fe720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860fec60 0x560486101050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560486058e00 con 0x5604860fec60 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560486001b10 con 0x5604860f5760 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f967e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560486054ea0 con 0x5604860f6080 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0edffb640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 0x7fe0e00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0effff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0effff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:54272/0 (socket says 172.21.15.73:54272) 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0effff640 1 -- 172.21.15.73:0/3644858868 learned_addr learned my addr 172.21.15.73:0/3644858868 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0effff640 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860fec60 0x560486101050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f4836640 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860f6080 0x5604860fe720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.043+0000 7fe0f5037640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 0x7fe0e00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 -- 172.21.15.73:0/3644858868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860f6080 msgr2=0x5604860fe720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860f6080 0x5604860fe720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 -- 172.21.15.73:0/3644858868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860fec60 msgr2=0x560486101050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0edffb640 1 -- 172.21.15.73:0/4251161422 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe0e80f0d40 con 0x7fe0f0073120 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860fec60 0x560486101050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 -- 172.21.15.73:0/3644858868 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560485e377d0 con 0x5604860f5760 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0effff640 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc005bc0 tx=0x7fe0dc005850 comp rx=0 tx=0).ready entity=mon.2 client_cookie=aa9cc3a0896e6a17 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0f5037640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 0x7fe0e00a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0e80ee2c0 tx=0x7fe0e80b8d20 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0daffd640 1 -- 172.21.15.73:0/3644858868 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc004020 con 0x5604860f5760 2024-03-09T00:11:32.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0daffd640 1 -- 172.21.15.73:0/3644858868 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0dc00fca0 con 0x5604860f5760 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.044+0000 7fe0daffd640 1 -- 172.21.15.73:0/3644858868 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc0039c0 con 0x5604860f5760 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- 172.21.15.73:0/3644858868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 msgr2=0x5604860f5b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc005bc0 tx=0x7fe0dc005850 comp rx=0 tx=0).stop 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- 172.21.15.73:0/3644858868 shutdown_connections 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860fec60 0x560486101050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860f6080 0x5604860fe720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3644858868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604860f5b40 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- 172.21.15.73:0/3644858868 >> 172.21.15.73:0/3644858868 conn(0x5604860efe90 msgr2=0x5604860f0270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- 172.21.15.73:0/3644858868 shutdown_connections 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- 172.21.15.73:0/3644858868 wait complete. 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 -- start start 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.045+0000 7fe0f967e740 1 Processor -- start 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f6080 0x5604860fb710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fbc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604861019f0 con 0x5604860f5760 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5604860150a0 con 0x5604860f6080 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560486101b70 con 0x5604860fec60 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f967e740 10 client.0 _ll_register_callbacks cb 0x560486014a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39334/0 (socket says 172.21.15.73:39334) 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 -- 172.21.15.73:0/522483244 learned_addr learned my addr 172.21.15.73:0/522483244 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:32.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0f4836640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f6080 0x5604860fb710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fbc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 -- 172.21.15.73:0/522483244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 msgr2=0x5604860fbc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fbc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 -- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f6080 msgr2=0x5604860fb710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f6080 0x5604860fb710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 -- 172.21.15.73:0/522483244 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560485d6d3f0 con 0x5604860f5760 2024-03-09T00:11:32.056 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0effff640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0058c0 tx=0x7fe0dc0035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1afd5f9a752a0f7f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.046+0000 7fe0cffff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc005050 con 0x5604860f5760 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe0dc028040 con 0x5604860f5760 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0dc00fca0 con 0x5604860f5760 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0dc034050 con 0x5604860f5760 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 10 client.16485 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560485d2d0f0 con 0x5604860f5760 2024-03-09T00:11:32.057 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0dc00fca0 con 0x5604860f5760 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe0dc01d030 con 0x5604860f5760 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 10 client.16485 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 10 client.16485 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 20 client.16485 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56048610e530 0x560486110920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 4 client.16485 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:11:32.058 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560486066340 con 0x56048610e530 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.047+0000 7fe0f4836640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56048610e530 0x560486110920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.048+0000 7fe0f4836640 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56048610e530 0x560486110920 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ee4f6835854b468c server_cookie=5ca9a542989c7404 in_seq=0 out_seq=0 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.048+0000 7fe0cffff640 10 client.16485 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0cffff640 1 -- 172.21.15.73:0/522483244 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560486066340 con 0x56048610e530 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0cffff640 10 client.16485 handle_command_reply: tid=0 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 client.16485 shutdown 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 msgr2=0x5604860fe380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.059 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0058c0 tx=0x7fe0dc0035f0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 shutdown_connections 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56048610e530 0x560486110920 crc :-1 s=READY pgs=127 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fbc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f6080 0x5604860fb710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 --2- 172.21.15.73:0/522483244 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f5760 0x5604860fe380 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 >> 172.21.15.73:0/522483244 conn(0x5604860efe90 msgr2=0x5604860f1be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0cffff640 0 client.16485 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:32.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 shutdown_connections 2024-03-09T00:11:32.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 1 -- 172.21.15.73:0/522483244 wait complete. 2024-03-09T00:11:32.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.049+0000 7fe0f967e740 20 client.16485 trim_cache size 0 max 16384 2024-03-09T00:11:32.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 Processor -- start 2024-03-09T00:11:32.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 -- start start 2024-03-09T00:11:32.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604861150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fde40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b4eb0 con 0x5604860f6080 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b7470 con 0x5604860f5760 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f967e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b54b0 con 0x5604860fec60 2024-03-09T00:11:32.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f4836640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f4836640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39350/0 (socket says 172.21.15.73:39350) 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0effff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604861150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0f4836640 1 -- 172.21.15.73:0/838579983 learned_addr learned my addr 172.21.15.73:0/838579983 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.067+0000 7fe0effff640 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fde40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 -- 172.21.15.73:0/838579983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 msgr2=0x5604860fde40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fde40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 -- 172.21.15.73:0/838579983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 msgr2=0x5604861150f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604861150f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 -- 172.21.15.73:0/838579983 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5604860ea330 con 0x5604860f6080 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0f4836640 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009730 tx=0x7fe0e4005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a65bcdec90ec43b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0da7fc640 1 -- 172.21.15.73:0/838579983 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4003d60 con 0x5604860f6080 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0da7fc640 1 -- 172.21.15.73:0/838579983 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0e401e870 con 0x5604860f6080 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.068+0000 7fe0da7fc640 1 -- 172.21.15.73:0/838579983 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0e4010e50 con 0x5604860f6080 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- 172.21.15.73:0/838579983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 msgr2=0x5604860fa6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 crypto rx=0x7fe0e4009730 tx=0x7fe0e4005f90 comp rx=0 tx=0).stop 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- 172.21.15.73:0/838579983 shutdown_connections 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x5604860fde40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x5604860fa6f0 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 --2- 172.21.15.73:0/838579983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x5604861150f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- 172.21.15.73:0/838579983 >> 172.21.15.73:0/838579983 conn(0x5604860efe90 msgr2=0x5604860f9fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- 172.21.15.73:0/838579983 shutdown_connections 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- 172.21.15.73:0/838579983 wait complete. 2024-03-09T00:11:32.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 -- start start 2024-03-09T00:11:32.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.069+0000 7fe0f967e740 1 Processor -- start 2024-03-09T00:11:32.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x560486106320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x560486102360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b5af0 con 0x5604860f6080 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b6e10 con 0x5604860f5760 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0f01b6150 con 0x5604860fec60 2024-03-09T00:11:32.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f967e740 10 client.0 _ll_register_callbacks cb 0x7fe0dc009e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:54298/0 (socket says 172.21.15.73:54298) 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 -- 172.21.15.73:0/3570290311 learned_addr learned my addr 172.21.15.73:0/3570290311 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x560486102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0f4836640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x560486106320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 -- 172.21.15.73:0/3570290311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 msgr2=0x560486102360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x560486102360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 -- 172.21.15.73:0/3570290311 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 msgr2=0x560486106320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x560486106320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.070+0000 7fe0effff640 1 -- 172.21.15.73:0/3570290311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560486102b60 con 0x5604860f5760 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0effff640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0059f0 tx=0x7fe0dc00f8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=60f4984930a30bcc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0cffff640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0dc009b10 con 0x5604860f5760 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe0e401e870 con 0x5604860f5760 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe0dc011030 con 0x5604860f5760 2024-03-09T00:11:32.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0dc045050 con 0x5604860f5760 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0f967e740 10 client.25799 fetch_fsmap learned FSMap version 20 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560486165510 con 0x5604860f5760 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.071+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0e401e870 con 0x5604860f5760 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe0dc01d070 con 0x5604860f5760 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 10 client.25799 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 10 client.25799 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:11:32.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 20 client.25799 populate_metadata read hostname 'smithi073' 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560486165bb0 0x560486167fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 4 client.25799 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560486060690 con 0x560486165bb0 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f4836640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560486165bb0 0x560486167fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0f4836640 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560486165bb0 0x560486167fd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b27ae95c7cfa747 server_cookie=d12ea715d6fff56b in_seq=0 out_seq=0 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.072+0000 7fe0cffff640 10 client.25799 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:32.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0cffff640 1 -- 172.21.15.73:0/3570290311 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x560486060690 con 0x560486165bb0 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0cffff640 10 client.25799 handle_command_reply: tid=0 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 client.25799 shutdown 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 msgr2=0x560486105de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fe0dc0059f0 tx=0x7fe0dc00f8c0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 shutdown_connections 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560486165bb0 0x560486167fd0 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5604860fec60 0x560486102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5604860f6080 0x560486106320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 --2- 172.21.15.73:0/3570290311 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5604860f5760 0x560486105de0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 >> 172.21.15.73:0/3570290311 conn(0x5604860efe90 msgr2=0x5604860fa870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0cffff640 0 client.25799 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 shutdown_connections 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 1 -- 172.21.15.73:0/3570290311 wait complete. 2024-03-09T00:11:32.080 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.073+0000 7fe0f967e740 20 client.25799 trim_cache size 0 max 16384 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2576 inodes in the stack)", 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "00e05d7d-ed88-4d6e-b3c3-03978523d737": { 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "00e05d7d-ed88-4d6e-b3c3-03978523d737", 2024-03-09T00:11:32.081 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:11:32.082 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:11:32.082 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:11:32.082 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:11:32.083 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 msgr2=0x7fe0e00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 0x7fe0e00a4e30 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fe0e80ee2c0 tx=0x7fe0e80b8d20 comp rx=0 tx=0).stop 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 msgr2=0x7fe0f006f120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 crypto rx=0x7fe0e802c700 tx=0x7fe0e8032f20 comp rx=0 tx=0).stop 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 shutdown_connections 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe0e00a2970 0x7fe0e00a4e30 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe0f0073120 0x7fe0f006f120 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.084 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe0f01054a0 0x7fe0f006eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 --2- 172.21.15.73:0/4251161422 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe0f01003d0 0x7fe0f0072be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:11:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 >> 172.21.15.73:0/4251161422 conn(0x7fe0f00772f0 msgr2=0x7fe0f00fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:11:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 shutdown_connections 2024-03-09T00:11:32.085 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:11:32.074+0000 7fe0f6a9a640 1 -- 172.21.15.73:0/4251161422 wait complete. 2024-03-09T00:11:32.085 INFO:tasks.cephfs.filesystem:scrub status for tag:00e05d7d-ed88-4d6e-b3c3-03978523d737 - {'path': '/', 'tag': '00e05d7d-ed88-4d6e-b3c3-03978523d737', 'options': 'recursive,force'} 2024-03-09T00:11:32.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:32 smithi073 ceph-mon[32309]: pgmap v143: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:32.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:32 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:11:32.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:32 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:32.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:32 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:11:32.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:32 smithi102 ceph-mon[36106]: pgmap v143: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:32.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:32 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:11:32.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:32 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:32.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:32 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:11:32.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:32 smithi179 ceph-mon[36637]: pgmap v143: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:32.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:32 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:11:32.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:32 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:32.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:32 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:11:33.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:33 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:33.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:33 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:33.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:33 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:11:34.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:34 smithi073 ceph-mon[32309]: pgmap v144: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 60 op/s 2024-03-09T00:11:34.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:34 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:11:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:34 smithi102 ceph-mon[36106]: pgmap v144: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 60 op/s 2024-03-09T00:11:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:34 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:11:34.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:34 smithi179 ceph-mon[36637]: pgmap v144: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 60 op/s 2024-03-09T00:11:34.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:34 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:11:36.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:36 smithi073 ceph-mon[32309]: pgmap v145: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 59 op/s 2024-03-09T00:11:36.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:36 smithi102 ceph-mon[36106]: pgmap v145: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 59 op/s 2024-03-09T00:11:36.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:36 smithi179 ceph-mon[36637]: pgmap v145: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 59 op/s 2024-03-09T00:11:38.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:38 smithi073 ceph-mon[32309]: pgmap v146: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 203 op/s 2024-03-09T00:11:38.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:38 smithi102 ceph-mon[36106]: pgmap v146: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 203 op/s 2024-03-09T00:11:38.582 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:38 smithi179 ceph-mon[36637]: pgmap v146: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 203 KiB/s rd, 203 op/s 2024-03-09T00:11:40.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:40 smithi073 ceph-mon[32309]: pgmap v147: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:40.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:40 smithi102 ceph-mon[36106]: pgmap v147: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:40.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:40 smithi179 ceph-mon[36637]: pgmap v147: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:42.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:42 smithi073 ceph-mon[32309]: pgmap v148: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:42.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:42 smithi102 ceph-mon[36106]: pgmap v148: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:42.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:42 smithi179 ceph-mon[36637]: pgmap v148: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:11:43.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:43 smithi073 ceph-mon[32309]: pgmap v149: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:11:43.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:43 smithi102 ceph-mon[36106]: pgmap v149: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:11:43.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:43 smithi179 ceph-mon[36637]: pgmap v149: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:11:46.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:46 smithi073 ceph-mon[32309]: pgmap v150: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 op/s 2024-03-09T00:11:46.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:46 smithi102 ceph-mon[36106]: pgmap v150: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 op/s 2024-03-09T00:11:46.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:46 smithi179 ceph-mon[36637]: pgmap v150: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 230 op/s 2024-03-09T00:11:48.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:48 smithi073 ceph-mon[32309]: pgmap v151: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 231 op/s 2024-03-09T00:11:48.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:48 smithi102 ceph-mon[36106]: pgmap v151: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 231 op/s 2024-03-09T00:11:48.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:48 smithi179 ceph-mon[36637]: pgmap v151: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 230 KiB/s rd, 231 op/s 2024-03-09T00:11:50.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:50 smithi073 ceph-mon[32309]: pgmap v152: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 86 op/s 2024-03-09T00:11:50.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:50 smithi102 ceph-mon[36106]: pgmap v152: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 86 op/s 2024-03-09T00:11:50.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:50 smithi179 ceph-mon[36637]: pgmap v152: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 86 op/s 2024-03-09T00:11:52.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:52 smithi073 ceph-mon[32309]: pgmap v153: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:52.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:52 smithi102 ceph-mon[36106]: pgmap v153: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:52.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:52 smithi179 ceph-mon[36637]: pgmap v153: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:54 smithi073 ceph-mon[32309]: pgmap v154: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:54.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:54 smithi102 ceph-mon[36106]: pgmap v154: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:54.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:54 smithi179 ceph-mon[36637]: pgmap v154: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:56.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:56 smithi073 ceph-mon[32309]: pgmap v155: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:56.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:56 smithi102 ceph-mon[36106]: pgmap v155: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:56.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:56 smithi179 ceph-mon[36637]: pgmap v155: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:11:58.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:11:58 smithi073 ceph-mon[32309]: pgmap v156: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:58.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:11:58 smithi102 ceph-mon[36106]: pgmap v156: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:11:58.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:11:58 smithi179 ceph-mon[36637]: pgmap v156: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:00.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:00 smithi073 ceph-mon[32309]: pgmap v157: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:00.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:00 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:12:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:00 smithi102 ceph-mon[36106]: pgmap v157: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:00.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:00 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:12:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:00 smithi179 ceph-mon[36637]: pgmap v157: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:00 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:12:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:12:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:12:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:12:01.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:12:01.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:12:01.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:12:01.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:12:01.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:12:01.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:12:02.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:12:02.087 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:12:02.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 -- 172.21.15.73:0/2500866292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8104920 msgr2=0x7fcfa8104d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 --2- 172.21.15.73:0/2500866292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8104920 0x7fcfa8104d00 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800aa10 tx=0x7fcf9802cf70 comp rx=0 tx=0).stop 2024-03-09T00:12:02.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 -- 172.21.15.73:0/2500866292 shutdown_connections 2024-03-09T00:12:02.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 --2- 172.21.15.73:0/2500866292 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa8100550 0x7fcfa810c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 --2- 172.21.15.73:0/2500866292 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa8100010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 --2- 172.21.15.73:0/2500866292 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8104920 0x7fcfa8104d00 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 -- 172.21.15.73:0/2500866292 >> 172.21.15.73:0/2500866292 conn(0x7fcfa8075530 msgr2=0x7fcfa8075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 -- 172.21.15.73:0/2500866292 shutdown_connections 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.207+0000 7fcfac922640 1 -- 172.21.15.73:0/2500866292 wait complete. 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 Processor -- start 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 -- start start 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa8104920 0x7fcfa81979a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f500 con 0x7fcfa8100550 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f380 con 0x7fcfa8104920 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.208+0000 7fcfac922640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa810f680 con 0x7fcfa80ffbb0 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33214/0 (socket says 172.21.15.73:33214) 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 -- 172.21.15.73:0/3816733052 learned_addr learned my addr 172.21.15.73:0/3816733052 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa8104920 0x7fcfa81979a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcfa659c640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 -- 172.21.15.73:0/3816733052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 msgr2=0x7fcfa819d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa819d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 -- 172.21.15.73:0/3816733052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa8104920 msgr2=0x7fcfa81979a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa8104920 0x7fcfa81979a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 -- 172.21.15.73:0/3816733052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfa8198120 con 0x7fcfa8100550 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcfa659c640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000b4a0 tx=0x7fcf9000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9f7fe640 1 -- 172.21.15.73:0/3816733052 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90014070 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfa8198410 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9f7fe640 1 -- 172.21.15.73:0/3816733052 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf90010af0 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcf9f7fe640 1 -- 172.21.15.73:0/3816733052 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90017d10 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.209+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfa81a4730 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.211+0000 7fcf9f7fe640 1 -- 172.21.15.73:0/3816733052 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcf90017070 con 0x7fcfa8100550 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 Processor -- start 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 -- start start 2024-03-09T00:12:02.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcf9f7fe640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 0x7fcf940a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395a760 0x56229395ab40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x562293963720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622938bde00 con 0x562293963c60 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x562293866b10 con 0x56229395a760 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622938b9ea0 con 0x56229395b080 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 0x7fcf940a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcf9f7fe640 1 -- 172.21.15.73:0/3816733052 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcf900ce4d0 con 0x7fcfa8100550 2024-03-09T00:12:02.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfa6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfa6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33216/0 (socket says 172.21.15.73:33216) 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/4065661348 learned_addr learned my addr 172.21.15.73:0/4065661348 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.212+0000 7fcfa659c640 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395a760 0x56229395ab40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x562293963720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/4065661348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 msgr2=0x562293963720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x562293963720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/4065661348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395a760 msgr2=0x56229395ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395a760 0x56229395ab40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/4065661348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56229369c7d0 con 0x562293963c60 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 0x7fcf940a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0013b50 tx=0x7fcfa000bd30 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0010860 tx=0x7fcfa001a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=42343f2d5f6f9483 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcf9cff9640 1 -- 172.21.15.73:0/4065661348 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0021070 con 0x562293963c60 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcf9cff9640 1 -- 172.21.15.73:0/4065661348 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfa0003e50 con 0x562293963c60 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.213+0000 7fcf9cff9640 1 -- 172.21.15.73:0/4065661348 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa00043e0 con 0x562293963c60 2024-03-09T00:12:02.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/4065661348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 msgr2=0x562293966050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 secure :-1 s=READY pgs=780 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0010860 tx=0x7fcfa001a000 comp rx=0 tx=0).stop 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/4065661348 shutdown_connections 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x562293966050 unknown :-1 s=CLOSED pgs=780 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x562293963720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/4065661348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395a760 0x56229395ab40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/4065661348 >> 172.21.15.73:0/4065661348 conn(0x562293954e90 msgr2=0x562293955270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/4065661348 shutdown_connections 2024-03-09T00:12:02.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/4065661348 wait complete. 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 -- start start 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.214+0000 7fcfaf5d8740 1 Processor -- start 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x5622939606f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293960c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622939669f0 con 0x56229395a760 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56229387a0a0 con 0x562293963c60 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x562293966b70 con 0x56229395b080 2024-03-09T00:12:02.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 10 client.0 _ll_register_callbacks cb 0x562293879a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33222/0 (socket says 172.21.15.73:33222) 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 -- 172.21.15.73:0/3654475913 learned_addr learned my addr 172.21.15.73:0/3654475913 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293960c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x5622939606f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 -- 172.21.15.73:0/3654475913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 msgr2=0x5622939606f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x5622939606f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 -- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 msgr2=0x562293960c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293960c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 -- 172.21.15.73:0/3654475913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622935d23f0 con 0x56229395a760 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfa659c640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800aa10 tx=0x7fcf98035c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5124586b0dc1a9c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcf8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf98046020 con 0x56229395a760 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcf98004040 con 0x56229395a760 2024-03-09T00:12:02.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.215+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcfa0003e50 con 0x56229395a760 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcf98051050 con 0x56229395a760 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 10 client.16503 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5622935920f0 con 0x56229395a760 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcfa0003e50 con 0x56229395a760 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcf9800b040 con 0x56229395a760 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 10 client.16503 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 10 client.16503 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 20 client.16503 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:02.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939736a0 0x562293975ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 4 client.16503 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5622938cb340 con 0x5622939736a0 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.216+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939736a0 0x562293975ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.217+0000 7fcf8dffb640 10 client.16503 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.217+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939736a0 0x562293975ab0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99bb6019d2597ec0 server_cookie=aac231393b5521d2 in_seq=0 out_seq=0 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.217+0000 7fcf8dffb640 1 -- 172.21.15.73:0/3654475913 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5622938cb340 con 0x5622939736a0 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.217+0000 7fcf8dffb640 10 client.16503 handle_command_reply: tid=0 2024-03-09T00:12:02.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.217+0000 7fcfaf5d8740 1 client.16503 shutdown 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 msgr2=0x562293963360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 crypto rx=0x7fcf9800aa10 tx=0x7fcf98035c20 comp rx=0 tx=0).stop 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 shutdown_connections 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939736a0 0x562293975ab0 crc :-1 s=READY pgs=129 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293960c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcf8dffb640 0 client.16503 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395b080 0x5622939606f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/3654475913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395a760 0x562293963360 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 >> 172.21.15.73:0/3654475913 conn(0x562293954e90 msgr2=0x562293956be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 shutdown_connections 2024-03-09T00:12:02.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/3654475913 wait complete. 2024-03-09T00:12:02.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.218+0000 7fcfaf5d8740 20 client.16503 trim_cache size 0 max 16384 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.232+0000 7fcfaf5d8740 1 Processor -- start 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.232+0000 7fcfaf5d8740 1 -- start start 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x56229395ffd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293963360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81ac870 con 0x56229395b080 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81aee30 con 0x562293963c60 2024-03-09T00:12:02.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81ace70 con 0x56229395a760 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33238/0 (socket says 172.21.15.73:33238) 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 -- 172.21.15.73:0/2448775594 learned_addr learned my addr 172.21.15.73:0/2448775594 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfa659c640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x56229395ffd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 -- 172.21.15.73:0/2448775594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 msgr2=0x56229395ffd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293963360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x56229395ffd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 -- 172.21.15.73:0/2448775594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 msgr2=0x562293963360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293963360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 -- 172.21.15.73:0/2448775594 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622939599c0 con 0x56229395b080 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcfa659c640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x56229395ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fcf90096d10 tx=0x7fcf90085040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=46966e9c62171c18 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf8ffff640 1 -- 172.21.15.73:0/2448775594 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf9001e040 con 0x56229395b080 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf8ffff640 1 -- 172.21.15.73:0/2448775594 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf9009ea00 con 0x56229395b080 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.233+0000 7fcf8ffff640 1 -- 172.21.15.73:0/2448775594 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf90010ca0 con 0x56229395b080 2024-03-09T00:12:02.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2448775594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 msgr2=0x562293962e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 secure :-1 s=READY pgs=782 cs=0 l=1 rev1=1 crypto rx=0x7fcf90096d10 tx=0x7fcf90085040 comp rx=0 tx=0).stop 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2448775594 shutdown_connections 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562293963c60 0x562293963360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56229395b080 0x562293962e20 unknown :-1 s=CLOSED pgs=782 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2448775594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x56229395ffd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2448775594 >> 172.21.15.73:0/2448775594 conn(0x562293954e90 msgr2=0x56229395bcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2448775594 shutdown_connections 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2448775594 wait complete. 2024-03-09T00:12:02.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- start start 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 Processor -- start 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x5622939698e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395b080 0x562293969e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81ad4b0 con 0x562293963c60 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81ae7d0 con 0x56229395b080 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfa81adb10 con 0x56229395a760 2024-03-09T00:12:02.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.234+0000 7fcfaf5d8740 10 client.0 _ll_register_callbacks cb 0x7fcf9802ed00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33240/0 (socket says 172.21.15.73:33240) 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/2526499984 learned_addr learned my addr 172.21.15.73:0/2526499984 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf9ffff640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395b080 0x562293969e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa659c640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x5622939698e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/2526499984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 msgr2=0x5622939698e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x5622939698e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395b080 msgr2=0x562293969e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395b080 0x562293969e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 -- 172.21.15.73:0/2526499984 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622939c9c50 con 0x562293963c60 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfa6d9d640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0010860 tx=0x7fcfa000b860 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1af71bfebec1f206 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf8dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcfa0021040 con 0x562293963c60 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fcfa0003b70 con 0x562293963c60 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcf9009ea00 con 0x562293963c60 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcfa000ee60 con 0x562293963c60 2024-03-09T00:12:02.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfaf5d8740 10 client.16509 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5622939caed0 con 0x562293963c60 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.235+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf9009ea00 con 0x562293963c60 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcfa0004590 con 0x562293963c60 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 10 client.16509 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 10 client.16509 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 20 client.16509 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939cb480 0x5622939cd8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 4 client.16509 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5622938c5690 con 0x5622939cb480 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfa659c640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939cb480 0x5622939cd8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcf8dffb640 10 client.16509 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfa659c640 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939cb480 0x5622939cd8a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8672e23ba6fc2d27 server_cookie=4b5265d9eff152e0 in_seq=0 out_seq=0 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcf8dffb640 1 -- 172.21.15.73:0/2526499984 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5622938c5690 con 0x5622939cb480 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcf8dffb640 10 client.16509 handle_command_reply: tid=0 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.236+0000 7fcfaf5d8740 1 client.16509 shutdown 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 msgr2=0x56229396a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0010860 tx=0x7fcfa000b860 comp rx=0 tx=0).stop 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 shutdown_connections 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5622939cb480 0x5622939cd8a0 crc :-1 s=READY pgs=130 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562293963c60 0x56229396a360 unknown :-1 s=CLOSED pgs=783 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcf8dffb640 0 client.16509 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56229395b080 0x562293969e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 --2- 172.21.15.73:0/2526499984 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56229395a760 0x5622939698e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 >> 172.21.15.73:0/2526499984 conn(0x562293954e90 msgr2=0x5622939562c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 shutdown_connections 2024-03-09T00:12:02.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 1 -- 172.21.15.73:0/2526499984 wait complete. 2024-03-09T00:12:02.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfaf5d8740 20 client.16509 trim_cache size 0 max 16384 2024-03-09T00:12:02.249 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:02.249 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:12:02.249 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:12:02.250 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:02.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 msgr2=0x7fcf940a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 0x7fcf940a4e30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fcfa0013b50 tx=0x7fcfa000bd30 comp rx=0 tx=0).stop 2024-03-09T00:12:02.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 msgr2=0x7fcfa819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 secure :-1 s=READY pgs=779 cs=0 l=1 rev1=1 crypto rx=0x7fcf9000b4a0 tx=0x7fcf9000b960 comp rx=0 tx=0).stop 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 shutdown_connections 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcfa8104920 0x7fcfa81979a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf940a2970 0x7fcf940a4e30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcfa8100550 0x7fcfa819dd30 unknown :-1 s=CLOSED pgs=779 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 --2- 172.21.15.73:0/3816733052 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcfa80ffbb0 0x7fcfa819d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 >> 172.21.15.73:0/3816733052 conn(0x7fcfa8075530 msgr2=0x7fcfa80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 shutdown_connections 2024-03-09T00:12:02.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.237+0000 7fcfac922640 1 -- 172.21.15.73:0/3816733052 wait complete. 2024-03-09T00:12:02.253 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:12:02.253 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:12:02.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 -- 172.21.15.73:0/2894427014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc620102560 msgr2=0x7fc620102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 --2- 172.21.15.73:0/2894427014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc620102560 0x7fc620102940 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 crypto rx=0x7fc610009900 tx=0x7fc61002bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:02.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 -- 172.21.15.73:0/2894427014 shutdown_connections 2024-03-09T00:12:02.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 --2- 172.21.15.73:0/2894427014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc62010b930 0x7fc62010ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 --2- 172.21.15.73:0/2894427014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 0x7fc62010b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 --2- 172.21.15.73:0/2894427014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc620102560 0x7fc620102940 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 -- 172.21.15.73:0/2894427014 >> 172.21.15.73:0/2894427014 conn(0x7fc6200772f0 msgr2=0x7fc620077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 -- 172.21.15.73:0/2894427014 shutdown_connections 2024-03-09T00:12:02.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 -- 172.21.15.73:0/2894427014 wait complete. 2024-03-09T00:12:02.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.358+0000 7fc6269f1640 1 Processor -- start 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 -- start start 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc620102560 0x7fc62019d8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 0x7fc62019dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620111710 con 0x7fc62010b930 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620111590 con 0x7fc620102e80 2024-03-09T00:12:02.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc6269f1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc620111890 con 0x7fc620102560 2024-03-09T00:12:02.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33248/0 (socket says 172.21.15.73:33248) 2024-03-09T00:12:02.364 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 -- 172.21.15.73:0/320830014 learned_addr learned my addr 172.21.15.73:0/320830014 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc61ffff640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc620102560 0x7fc62019d8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc61f7fe640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 0x7fc62019dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 -- 172.21.15.73:0/320830014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc620102560 msgr2=0x7fc62019d8a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc620102560 0x7fc62019d8a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 -- 172.21.15.73:0/320830014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 msgr2=0x7fc62019dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 0x7fc62019dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.366 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 -- 172.21.15.73:0/320830014 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6201981d0 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.359+0000 7fc624f8e640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc61800cad0 tx=0x7fc61800cf90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.360+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800ae20 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.360+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc618005020 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.360+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6201984c0 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.360+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc61800f6f0 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.360+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6201a4760 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.361+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6200680e0 con 0x7fc62010b930 2024-03-09T00:12:02.367 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.362+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc6180051d0 con 0x7fc62010b930 2024-03-09T00:12:02.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.362+0000 7fc61d7fa640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 0x7fc6080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.362+0000 7fc61ffff640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 0x7fc6080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.362+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fc6180cf580 con 0x7fc62010b930 2024-03-09T00:12:02.368 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.363+0000 7fc61ffff640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 0x7fc6080a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc610009230 tx=0x7fc610005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.369 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.367+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc618014030 con 0x7fc62010b930 2024-03-09T00:12:02.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:02 smithi073 ceph-mon[32309]: pgmap v158: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:02.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:02 smithi102 ceph-mon[36106]: pgmap v158: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:02.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.578+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc620106a20 con 0x7fc62010b930 2024-03-09T00:12:02.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.579+0000 7fc61d7fa640 1 -- 172.21.15.73:0/320830014 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fc61809bf50 con 0x7fc62010b930 2024-03-09T00:12:02.576 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:12:02.576 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:12:02.576 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:12:02.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.581+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 msgr2=0x7fc6080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.581+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 0x7fc6080a4e30 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fc610009230 tx=0x7fc610005fd0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.581+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 msgr2=0x7fc620197a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.581+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 crypto rx=0x7fc61800cad0 tx=0x7fc61800cf90 comp rx=0 tx=0).stop 2024-03-09T00:12:02.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 shutdown_connections 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc6080a2970 0x7fc6080a4e30 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc62010b930 0x7fc620197a50 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc620102e80 0x7fc62019dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 --2- 172.21.15.73:0/320830014 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc620102560 0x7fc62019d8a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 >> 172.21.15.73:0/320830014 conn(0x7fc6200772f0 msgr2=0x7fc6200fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 shutdown_connections 2024-03-09T00:12:02.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.582+0000 7fc6269f1640 1 -- 172.21.15.73:0/320830014 wait complete. 2024-03-09T00:12:02.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:02 smithi179 ceph-mon[36637]: pgmap v158: 129 pgs: 129 active+clean; 35 MiB data, 702 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:02.589 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:12:02.589 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:12:02.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 -- 172.21.15.73:0/3316936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 msgr2=0x7f69041008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 --2- 172.21.15.73:0/3316936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f69041008c0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 crypto rx=0x7f6900009900 tx=0x7f690002bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:02.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 -- 172.21.15.73:0/3316936 shutdown_connections 2024-03-09T00:12:02.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 --2- 172.21.15.73:0/3316936 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904100e00 0x7f690410cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 --2- 172.21.15.73:0/3316936 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f69041008c0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 --2- 172.21.15.73:0/3316936 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904106280 0x7f6904106660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 -- 172.21.15.73:0/3316936 >> 172.21.15.73:0/3316936 conn(0x7f6904076f90 msgr2=0x7f69040773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 -- 172.21.15.73:0/3316936 shutdown_connections 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.697+0000 7f690c6cd640 1 -- 172.21.15.73:0/3316936 wait complete. 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 Processor -- start 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 -- start start 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904100e00 0x7f6904198f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904106280 0x7f69041a3aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041105a0 con 0x7f6904100440 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904110420 con 0x7f6904106280 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690c6cd640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6904110720 con 0x7f6904100e00 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33266/0 (socket says 172.21.15.73:33266) 2024-03-09T00:12:02.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 -- 172.21.15.73:0/4260112913 learned_addr learned my addr 172.21.15.73:0/4260112913 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690ac6a640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904106280 0x7f69041a3aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f6909c68640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904100e00 0x7f6904198f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 -- 172.21.15.73:0/4260112913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904100e00 msgr2=0x7f6904198f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904100e00 0x7f6904198f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 -- 172.21.15.73:0/4260112913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904106280 msgr2=0x7f69041a3aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904106280 0x7f69041a3aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.698+0000 7f690a469640 1 -- 172.21.15.73:0/4260112913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69041a3fe0 con 0x7f6904100440 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f690a469640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f68f800d770 tx=0x7f68f800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f68f77fe640 1 -- 172.21.15.73:0/4260112913 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8017070 con 0x7f6904100440 2024-03-09T00:12:02.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f68f77fe640 1 -- 172.21.15.73:0/4260112913 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68f8005500 con 0x7f6904100440 2024-03-09T00:12:02.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69041a41b0 con 0x7f6904100440 2024-03-09T00:12:02.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f68f77fe640 1 -- 172.21.15.73:0/4260112913 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68f8004e00 con 0x7f6904100440 2024-03-09T00:12:02.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.699+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69041a4680 con 0x7f6904100440 2024-03-09T00:12:02.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.701+0000 7f68f77fe640 1 -- 172.21.15.73:0/4260112913 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f68f8004fa0 con 0x7f6904100440 2024-03-09T00:12:02.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.701+0000 7f690f382740 1 Processor -- start 2024-03-09T00:12:02.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.701+0000 7f690f382740 1 -- start start 2024-03-09T00:12:02.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.701+0000 7f68f77fe640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 0x7f68f00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.701+0000 7f690f382740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed24580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690f382740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690f382740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed2fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690f382740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804eceb360 con 0x55804ed24ac0 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690f382740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804ec87ec0 con 0x55804ed2d6a0 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690f382740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804ec30d30 con 0x55804ed241a0 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690ac6a640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 0x7f68f00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f68f77fe640 1 -- 172.21.15.73:0/4260112913 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f68f80ceb10 con 0x7f6904100440 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33274/0 (socket says 172.21.15.73:33274) 2024-03-09T00:12:02.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 -- 172.21.15.73:0/1367385403 learned_addr learned my addr 172.21.15.73:0/1367385403 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690ac6a640 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed2fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690a469640 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed24580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 -- 172.21.15.73:0/1367385403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 msgr2=0x55804ed24580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed24580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 -- 172.21.15.73:0/1367385403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 msgr2=0x55804ed2fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed2fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 -- 172.21.15.73:0/1367385403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55804ec2fbc0 con 0x55804ed24ac0 2024-03-09T00:12:02.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f690ac6a640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 0x7f68f00a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc014bf0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f6909c68640 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f690002c170 tx=0x7f690002c5d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8104080fdb4c92dc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f68f4ff9640 1 -- 172.21.15.73:0/1367385403 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900042070 con 0x55804ed24ac0 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f68f4ff9640 1 -- 172.21.15.73:0/1367385403 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f690002ca80 con 0x55804ed24ac0 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.702+0000 7f68f4ff9640 1 -- 172.21.15.73:0/1367385403 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900004f20 con 0x55804ed24ac0 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- 172.21.15.73:0/1367385403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 msgr2=0x55804ed2d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 crypto rx=0x7f690002c170 tx=0x7f690002c5d0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- 172.21.15.73:0/1367385403 shutdown_connections 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed2fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed2d160 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 --2- 172.21.15.73:0/1367385403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed24580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- 172.21.15.73:0/1367385403 >> 172.21.15.73:0/1367385403 conn(0x55804ed1e8d0 msgr2=0x55804ed1ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- 172.21.15.73:0/1367385403 shutdown_connections 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- 172.21.15.73:0/1367385403 wait complete. 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.703+0000 7f690f382740 1 -- start start 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 Processor -- start 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed24ac0 0x55804ed2ac10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804ec264a0 con 0x55804ed2d6a0 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804ec83f60 con 0x55804ed241a0 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55804ed30430 con 0x55804ed24ac0 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 10 client.0 _ll_register_callbacks cb 0x55804ec81d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33286/0 (socket says 172.21.15.73:33286) 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 -- 172.21.15.73:0/521827858 learned_addr learned my addr 172.21.15.73:0/521827858 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690a469640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f6909c68640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed24ac0 0x55804ed2ac10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 -- 172.21.15.73:0/521827858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed24ac0 msgr2=0x55804ed2ac10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed24ac0 0x55804ed2ac10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 -- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 msgr2=0x55804ed2a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 -- 172.21.15.73:0/521827858 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55804e99c3f0 con 0x55804ed2d6a0 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690a469640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690ac6a640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f68fc012fb0 tx=0x7f68fc004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ceb92cda037de98d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f68edffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68fc025070 con 0x55804ed2d6a0 2024-03-09T00:12:02.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f68fc021420 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.704+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f690002ca80 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68fc031050 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 10 client.16527 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55804ea667d0 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f690002ca80 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f68fc00f040 con 0x55804ed2d6a0 2024-03-09T00:12:02.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 10 client.16527 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 10 client.16527 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 20 client.16527 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed3d830 0x55804ed3fc60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 4 client.16527 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55804ec95400 con 0x55804ed3d830 2024-03-09T00:12:02.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.705+0000 7f690a469640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed3d830 0x55804ed3fc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f68edffb640 10 client.16527 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690a469640 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed3d830 0x55804ed3fc60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0044b2593b4ab14 server_cookie=3e760963f4eda646 in_seq=0 out_seq=0 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f68edffb640 1 -- 172.21.15.73:0/521827858 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55804ec95400 con 0x55804ed3d830 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f68edffb640 10 client.16527 handle_command_reply: tid=0 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 client.16527 shutdown 2024-03-09T00:12:02.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 msgr2=0x55804ed2b150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 crypto rx=0x7f68fc012fb0 tx=0x7f68fc004ab0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 shutdown_connections 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed3d830 0x55804ed3fc60 crc :-1 s=READY pgs=131 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed2d6a0 0x55804ed2b150 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed24ac0 0x55804ed2ac10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 --2- 172.21.15.73:0/521827858 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f68edffb640 0 client.16527 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 >> 172.21.15.73:0/521827858 conn(0x55804ed1e8d0 msgr2=0x55804ed20620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 shutdown_connections 2024-03-09T00:12:02.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.706+0000 7f690f382740 1 -- 172.21.15.73:0/521827858 wait complete. 2024-03-09T00:12:02.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.707+0000 7f690f382740 20 client.16527 trim_cache size 0 max 16384 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 Processor -- start 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 -- start start 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2ce00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed2d6a0 0x55804ed27930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.728 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041ac710 con 0x55804ed24ac0 2024-03-09T00:12:02.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041aecd0 con 0x55804ed241a0 2024-03-09T00:12:02.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690f382740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041acd10 con 0x55804ed2d6a0 2024-03-09T00:12:02.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33294/0 (socket says 172.21.15.73:33294) 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 -- 172.21.15.73:0/2711397285 learned_addr learned my addr 172.21.15.73:0/2711397285 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690ac6a640 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed2d6a0 0x55804ed27930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f690a469640 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2ce00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 -- 172.21.15.73:0/2711397285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed2d6a0 msgr2=0x55804ed27930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed2d6a0 0x55804ed27930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 -- 172.21.15.73:0/2711397285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 msgr2=0x55804ed2ce00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2ce00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.718+0000 7f6909c68640 1 -- 172.21.15.73:0/2711397285 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55804ed248f0 con 0x55804ed24ac0 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f6909c68640 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f690002c170 tx=0x7f690002cac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1ab8475085c32bd4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f68effff640 1 -- 172.21.15.73:0/2711397285 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900042040 con 0x55804ed24ac0 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f68effff640 1 -- 172.21.15.73:0/2711397285 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69000046c0 con 0x55804ed24ac0 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f68effff640 1 -- 172.21.15.73:0/2711397285 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900005160 con 0x55804ed24ac0 2024-03-09T00:12:02.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- 172.21.15.73:0/2711397285 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 msgr2=0x55804ed296b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 crypto rx=0x7f690002c170 tx=0x7f690002cac0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- 172.21.15.73:0/2711397285 shutdown_connections 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed2d6a0 0x55804ed27930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed296b0 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 --2- 172.21.15.73:0/2711397285 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed241a0 0x55804ed2ce00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- 172.21.15.73:0/2711397285 >> 172.21.15.73:0/2711397285 conn(0x55804ed1e8d0 msgr2=0x55804ed28fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- 172.21.15.73:0/2711397285 shutdown_connections 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- 172.21.15.73:0/2711397285 wait complete. 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.719+0000 7f690f382740 1 -- start start 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 Processor -- start 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed33740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed93d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041ad350 con 0x55804ed24ac0 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041ae670 con 0x55804ed2d6a0 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69041ad9b0 con 0x55804ed241a0 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 10 client.0 _ll_register_callbacks cb 0x7f6900005dd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33308/0 (socket says 172.21.15.73:33308) 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 -- 172.21.15.73:0/3955506144 learned_addr learned my addr 172.21.15.73:0/3955506144 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690a469640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed33740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 -- 172.21.15.73:0/3955506144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 msgr2=0x55804ed33740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed33740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 -- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 msgr2=0x55804ed93d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed93d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 -- 172.21.15.73:0/3955506144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55804ed94290 con 0x55804ed24ac0 2024-03-09T00:12:02.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690a469640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f6909c68640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f69000096c0 tx=0x7f6900005e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a726f2fb3fa10a8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f68edffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6900052070 con 0x55804ed24ac0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6900003a00 con 0x55804ed24ac0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.720+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69000046c0 con 0x55804ed24ac0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6900060050 con 0x55804ed24ac0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 10 client.16533 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55804ed95570 con 0x55804ed24ac0 2024-03-09T00:12:02.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69000046c0 con 0x55804ed24ac0 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6900034440 con 0x55804ed24ac0 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 10 client.16533 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 10 client.16533 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 20 client.16533 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed95840 0x55804ed97c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 4 client.16533 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55804ec8f750 con 0x55804ed95840 2024-03-09T00:12:02.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.721+0000 7f690a469640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed95840 0x55804ed97c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f68edffb640 10 client.16533 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690a469640 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed95840 0x55804ed97c30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e5cde45a4961cddf server_cookie=da43effcdb754682 in_seq=0 out_seq=0 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f68edffb640 1 -- 172.21.15.73:0/3955506144 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x7f68f809e020 con 0x55804ed95840 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f68edffb640 10 client.16533 handle_command_reply: tid=0 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 client.16533 shutdown 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 msgr2=0x55804ed33c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 crypto rx=0x7f69000096c0 tx=0x7f6900005e80 comp rx=0 tx=0).stop 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 shutdown_connections 2024-03-09T00:12:02.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55804ed95840 0x55804ed97c30 crc :-1 s=READY pgs=132 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55804ed2d6a0 0x55804ed93d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f68edffb640 0 client.16533 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55804ed24ac0 0x55804ed33c80 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 --2- 172.21.15.73:0/3955506144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55804ed241a0 0x55804ed33740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 >> 172.21.15.73:0/3955506144 conn(0x55804ed1e8d0 msgr2=0x55804ed29830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 shutdown_connections 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 1 -- 172.21.15.73:0/3955506144 wait complete. 2024-03-09T00:12:02.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690f382740 20 client.16533 trim_cache size 0 max 16384 2024-03-09T00:12:02.740 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:12:02.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 msgr2=0x7f68f00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 0x7f68f00a4e30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f68fc005fd0 tx=0x7f68fc014bf0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 msgr2=0x7f6904198a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.722+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 secure :-1 s=READY pgs=787 cs=0 l=1 rev1=1 crypto rx=0x7f68f800d770 tx=0x7f68f800dc30 comp rx=0 tx=0).stop 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 shutdown_connections 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6904106280 0x7f69041a3aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6904100e00 0x7f6904198f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f68f00a2970 0x7f68f00a4e30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 --2- 172.21.15.73:0/4260112913 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6904100440 0x7f6904198a20 unknown :-1 s=CLOSED pgs=787 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:02.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 >> 172.21.15.73:0/4260112913 conn(0x7f6904076f90 msgr2=0x7f69040ff1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:02.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 shutdown_connections 2024-03-09T00:12:02.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:02.723+0000 7f690c6cd640 1 -- 172.21.15.73:0/4260112913 wait complete. 2024-03-09T00:12:03.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:03 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/320830014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:03.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:03 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/320830014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:03.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:03 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/320830014' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:03.743 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:12:03.744 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:12:03.744 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:12:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 -- 172.21.15.73:0/1283730474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 msgr2=0x7fa9dc10dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1283730474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc10dda0 secure :-1 s=READY pgs=792 cs=0 l=1 rev1=1 crypto rx=0x7fa9d4009ca0 tx=0x7fa9d402c180 comp rx=0 tx=0).stop 2024-03-09T00:12:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 -- 172.21.15.73:0/1283730474 shutdown_connections 2024-03-09T00:12:03.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1283730474 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc10dda0 unknown :-1 s=CLOSED pgs=792 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1283730474 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 0x7fa9dc10b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1283730474 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 0x7fa9dc102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 -- 172.21.15.73:0/1283730474 >> 172.21.15.73:0/1283730474 conn(0x7fa9dc077350 msgr2=0x7fa9dc077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 -- 172.21.15.73:0/1283730474 shutdown_connections 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.865+0000 7fa9e260b640 1 -- 172.21.15.73:0/1283730474 wait complete. 2024-03-09T00:12:03.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 Processor -- start 2024-03-09T00:12:03.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 -- start start 2024-03-09T00:12:03.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 0x7fa9dc1001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 0x7fa9dc1006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1116e0 con 0x7fa9dc10b930 2024-03-09T00:12:03.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc111560 con 0x7fa9dc102e80 2024-03-09T00:12:03.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e260b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc111860 con 0x7fa9dc102560 2024-03-09T00:12:03.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33320/0 (socket says 172.21.15.73:33320) 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 -- 172.21.15.73:0/1851774085 learned_addr learned my addr 172.21.15.73:0/1851774085 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 0x7fa9dc1001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 0x7fa9dc1006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 -- 172.21.15.73:0/1851774085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 msgr2=0x7fa9dc1001b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 0x7fa9dc1001b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 -- 172.21.15.73:0/1851774085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 msgr2=0x7fa9dc1006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 0x7fa9dc1006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.866+0000 7fa9e0ba8640 1 -- 172.21.15.73:0/1851774085 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9dc1a60a0 con 0x7fa9dc10b930 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9e0ba8640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa9d402c790 tx=0x7fa9d4035ed0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9d97fa640 1 -- 172.21.15.73:0/1851774085 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d4039040 con 0x7fa9dc10b930 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9dc1a6330 con 0x7fa9dc10b930 2024-03-09T00:12:03.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9d97fa640 1 -- 172.21.15.73:0/1851774085 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9d4044070 con 0x7fa9dc10b930 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9d97fa640 1 -- 172.21.15.73:0/1851774085 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d40329f0 con 0x7fa9dc10b930 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.867+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9dc1a6820 con 0x7fa9dc10b930 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9d97fa640 1 -- 172.21.15.73:0/1851774085 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa9d4032b90 con 0x7fa9dc10b930 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 Processor -- start 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 -- start start 2024-03-09T00:12:03.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1419d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa141c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa1374b40 con 0x564fa1410d80 2024-03-09T00:12:03.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa131dd80 con 0x564fa141a280 2024-03-09T00:12:03.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa1370be0 con 0x564fa14116a0 2024-03-09T00:12:03.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9d97fa640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 0x7fa9c40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33336/0 (socket says 172.21.15.73:33336) 2024-03-09T00:12:03.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9db7fe640 1 -- 172.21.15.73:0/3394604438 learned_addr learned my addr 172.21.15.73:0/3394604438 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:03.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1419d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9e0ba8640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 0x7fa9c40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9db7fe640 1 -- 172.21.15.73:0/3394604438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 msgr2=0x564fa1419d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.870+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1419d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9db7fe640 1 -- 172.21.15.73:0/3394604438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 msgr2=0x564fa141c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa141c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9db7fe640 1 -- 172.21.15.73:0/3394604438 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fa11517e0 con 0x564fa1410d80 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9d97fa640 1 -- 172.21.15.73:0/1851774085 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa9d40f1650 con 0x7fa9dc10b930 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009c50 tx=0x7fa9cc0137d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5412e5c2790a4e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9c27fc640 1 -- 172.21.15.73:0/3394604438 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc013070 con 0x564fa1410d80 2024-03-09T00:12:03.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9e0ba8640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 0x7fa9c40a4e60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40eebd0 tx=0x7fa9d4005df0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9c27fc640 1 -- 172.21.15.73:0/3394604438 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc017030 con 0x564fa1410d80 2024-03-09T00:12:03.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9c27fc640 1 -- 172.21.15.73:0/3394604438 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc01b610 con 0x564fa1410d80 2024-03-09T00:12:03.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9e51ed740 1 -- 172.21.15.73:0/3394604438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 msgr2=0x564fa1411160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.871+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc009c50 tx=0x7fa9cc0137d0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- 172.21.15.73:0/3394604438 shutdown_connections 2024-03-09T00:12:03.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa141c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1419d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/3394604438 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa1411160 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- 172.21.15.73:0/3394604438 >> 172.21.15.73:0/3394604438 conn(0x564fa140b4b0 msgr2=0x564fa140b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- 172.21.15.73:0/3394604438 shutdown_connections 2024-03-09T00:12:03.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- 172.21.15.73:0/3394604438 wait complete. 2024-03-09T00:12:03.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- start start 2024-03-09T00:12:03.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 Processor -- start 2024-03-09T00:12:03.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa1410d80 0x564fa1419ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1417030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa141d070 con 0x564fa141a280 2024-03-09T00:12:03.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa13311c0 con 0x564fa1410d80 2024-03-09T00:12:03.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fa141d1f0 con 0x564fa14116a0 2024-03-09T00:12:03.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.872+0000 7fa9e51ed740 10 client.0 _ll_register_callbacks cb 0x564fa13207a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:03.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33348/0 (socket says 172.21.15.73:33348) 2024-03-09T00:12:03.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 -- 172.21.15.73:0/175251735 learned_addr learned my addr 172.21.15.73:0/175251735 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:03.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa1410d80 0x564fa1419ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1417030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 -- 172.21.15.73:0/175251735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 msgr2=0x564fa1417030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1417030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 -- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa1410d80 msgr2=0x564fa1419ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa1410d80 0x564fa1419ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 -- 172.21.15.73:0/175251735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fa11b1f70 con 0x564fa141a280 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1417030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc006010 tx=0x7fa9cc00fe30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79707959a89de6d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:03.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0130f0 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa9cc011440 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9cc017030 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9cc032050 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9e51ed740 10 client.16545 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564fa1049170 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.873+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9cc017030 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa9cc01c5b0 con 0x564fa141a280 2024-03-09T00:12:03.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 10 client.16545 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 10 client.16545 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 20 client.16545 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa142a390 0x564fa142c780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 4 client.16545 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fa134a960 con 0x564fa142a390 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa142a390 0x564fa142c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9bb7fe640 10 client.16545 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.874+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa142a390 0x564fa142c780 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1c67744c266b2267 server_cookie=13bc2d71ffa6ced9 in_seq=0 out_seq=0 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/175251735 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564fa134a960 con 0x564fa142a390 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9bb7fe640 10 client.16545 handle_command_reply: tid=0 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 client.16545 shutdown 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 msgr2=0x564fa1417640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 secure :-1 s=READY pgs=795 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc006010 tx=0x7fa9cc00fe30 comp rx=0 tx=0).stop 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 shutdown_connections 2024-03-09T00:12:03.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa142a390 0x564fa142c780 crc :-1 s=READY pgs=133 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa141a280 0x564fa1417640 unknown :-1 s=CLOSED pgs=795 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa14116a0 0x564fa1417030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/175251735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa1410d80 0x564fa1419ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9bb7fe640 0 client.16545 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 >> 172.21.15.73:0/175251735 conn(0x564fa140b4b0 msgr2=0x564fa140d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 shutdown_connections 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 1 -- 172.21.15.73:0/175251735 wait complete. 2024-03-09T00:12:03.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.875+0000 7fa9e51ed740 20 client.16545 trim_cache size 0 max 16384 2024-03-09T00:12:03.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 Processor -- start 2024-03-09T00:12:03.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 -- start start 2024-03-09T00:12:03.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa1410d80 0x564fa1419760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa1414290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1ae850 con 0x564fa14116a0 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1b0e10 con 0x564fa141a280 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1aee50 con 0x564fa1410d80 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33350/0 (socket says 172.21.15.73:33350) 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 -- 172.21.15.73:0/383761644 learned_addr learned my addr 172.21.15.73:0/383761644 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:03.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa1414290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa1410d80 0x564fa1419760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 -- 172.21.15.73:0/383761644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa1410d80 msgr2=0x564fa1419760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa1410d80 0x564fa1419760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 -- 172.21.15.73:0/383761644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 msgr2=0x564fa1414290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa1414290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 -- 172.21.15.73:0/383761644 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fa14114d0 con 0x564fa14116a0 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa1414290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x564fa1406df0 tx=0x7fa9d0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ba89321620662dc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9c1ffb640 1 -- 172.21.15.73:0/383761644 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0020070 con 0x564fa14116a0 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9c1ffb640 1 -- 172.21.15.73:0/383761644 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9d001c880 con 0x564fa14116a0 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.891+0000 7fa9c1ffb640 1 -- 172.21.15.73:0/383761644 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9d0004a00 con 0x564fa14116a0 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- 172.21.15.73:0/383761644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 msgr2=0x564fa1419ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 secure :-1 s=READY pgs=796 cs=0 l=1 rev1=1 crypto rx=0x564fa1406df0 tx=0x7fa9d0011810 comp rx=0 tx=0).stop 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- 172.21.15.73:0/383761644 shutdown_connections 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa141a280 0x564fa1414290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa14116a0 0x564fa1419ca0 unknown :-1 s=CLOSED pgs=796 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/383761644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa1410d80 0x564fa1419760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- 172.21.15.73:0/383761644 >> 172.21.15.73:0/383761644 conn(0x564fa140b4b0 msgr2=0x564fa1416ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- 172.21.15.73:0/383761644 shutdown_connections 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- 172.21.15.73:0/383761644 wait complete. 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 -- start start 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.892+0000 7fa9e51ed740 1 Processor -- start 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa14116a0 0x564fa141dd30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa141a280 0x564fa141e420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1af490 con 0x564fa1410d80 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1b07b0 con 0x564fa14116a0 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9dc1afaf0 con 0x564fa141a280 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 10 client.0 _ll_register_callbacks cb 0x7fa9cc009650 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33356/0 (socket says 172.21.15.73:33356) 2024-03-09T00:12:03.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 -- 172.21.15.73:0/1847410910 learned_addr learned my addr 172.21.15.73:0/1847410910 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa14116a0 0x564fa141dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 -- 172.21.15.73:0/1847410910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa141a280 msgr2=0x564fa141e420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa141a280 0x564fa141e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 -- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa14116a0 msgr2=0x564fa141dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa14116a0 0x564fa141dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 -- 172.21.15.73:0/1847410910 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fa14806d0 con 0x564fa1410d80 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9db7fe640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc01ca80 tx=0x7fa9cc01cab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f9b0a1f392bebb server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:03.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9bb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa9cc0130f0 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa9cc00f750 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa9d001c880 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa9cc00f950 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.893+0000 7fa9e51ed740 10 client.16551 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564fa1481820 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa9d001c880 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa9cc0035f0 con 0x564fa1410d80 2024-03-09T00:12:03.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 10 client.16551 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 10 client.16551 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 20 client.16551 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa1481ef0 0x564fa1484310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 4 client.16551 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x564fa1336080 con 0x564fa1481ef0 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa1481ef0 0x564fa1484310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9bb7fe640 10 client.16551 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:03.910 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.894+0000 7fa9dbfff640 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa1481ef0 0x564fa1484310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a36a40351b946322 server_cookie=a4ce67cec2cb636e in_seq=0 out_seq=0 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.895+0000 7fa9bb7fe640 1 -- 172.21.15.73:0/1847410910 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x564fa1336080 con 0x564fa1481ef0 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.895+0000 7fa9bb7fe640 10 client.16551 handle_command_reply: tid=0 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.895+0000 7fa9e51ed740 1 client.16551 shutdown 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 msgr2=0x564fa14219b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 crypto rx=0x7fa9cc01ca80 tx=0x7fa9cc01cab0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 shutdown_connections 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564fa1481ef0 0x564fa1484310 crc :-1 s=READY pgs=134 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564fa141a280 0x564fa141e420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.911 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564fa14116a0 0x564fa141dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 --2- 172.21.15.73:0/1847410910 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564fa1410d80 0x564fa14219b0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9bb7fe640 0 client.16551 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 >> 172.21.15.73:0/1847410910 conn(0x564fa140b4b0 msgr2=0x564fa1415900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 shutdown_connections 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 1 -- 172.21.15.73:0/1847410910 wait complete. 2024-03-09T00:12:03.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e51ed740 20 client.16551 trim_cache size 0 max 16384 2024-03-09T00:12:03.913 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:03.913 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:12:03.913 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "3da3df6c-042b-4abc-9499-61ed04ab646d", 2024-03-09T00:12:03.913 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:12:03.913 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 msgr2=0x7fa9c40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 0x7fa9c40a4e60 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fa9d40eebd0 tx=0x7fa9d4005df0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 msgr2=0x7fa9dc100c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 crypto rx=0x7fa9d402c790 tx=0x7fa9d4035ed0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 shutdown_connections 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa9c40a29a0 0x7fa9c40a4e60 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.915 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa9dc10b930 0x7fa9dc100c30 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa9dc102e80 0x7fa9dc1006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.896+0000 7fa9e260b640 1 --2- 172.21.15.73:0/1851774085 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa9dc102560 0x7fa9dc1001b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:03.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.897+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 >> 172.21.15.73:0/1851774085 conn(0x7fa9dc077350 msgr2=0x7fa9dc0fd340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:03.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.897+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 shutdown_connections 2024-03-09T00:12:03.916 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:03.897+0000 7fa9e260b640 1 -- 172.21.15.73:0/1851774085 wait complete. 2024-03-09T00:12:03.917 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:12:03.917 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:12:04.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 -- 172.21.15.73:0/1911961025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 msgr2=0x7f96001007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 --2- 172.21.15.73:0/1911961025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f96001007b0 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 crypto rx=0x7f95f0009900 tx=0x7f95f002bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:04.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 -- 172.21.15.73:0/1911961025 shutdown_connections 2024-03-09T00:12:04.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 --2- 172.21.15.73:0/1911961025 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001054a0 0x7f960010d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 --2- 172.21.15.73:0/1911961025 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9600100d80 0x7f9600104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 --2- 172.21.15.73:0/1911961025 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f96001007b0 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 -- 172.21.15.73:0/1911961025 >> 172.21.15.73:0/1911961025 conn(0x7f96000772f0 msgr2=0x7f9600077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 -- 172.21.15.73:0/1911961025 shutdown_connections 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.030+0000 7f9605427640 1 -- 172.21.15.73:0/1911961025 wait complete. 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 Processor -- start 2024-03-09T00:12:04.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 -- start start 2024-03-09T00:12:04.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96001054a0 0x7f9600198f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001994d0 0x7f96001a3a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9600110580 con 0x7f96001003d0 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9600110400 con 0x7f96001054a0 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f9605427640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9600110700 con 0x7f96001994d0 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95fe7fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96001054a0 0x7f9600198f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.032 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33372/0 (socket says 172.21.15.73:33372) 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 -- 172.21.15.73:0/3738761461 learned_addr learned my addr 172.21.15.73:0/3738761461 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001994d0 0x7f96001a3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 -- 172.21.15.73:0/3738761461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001994d0 msgr2=0x7f96001a3a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001994d0 0x7f96001a3a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 -- 172.21.15.73:0/3738761461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96001054a0 msgr2=0x7f9600198f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96001054a0 0x7f9600198f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.031+0000 7f95feffd640 1 -- 172.21.15.73:0/3738761461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96001a4250 con 0x7f96001003d0 2024-03-09T00:12:04.033 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f95feffd640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f95f002c240 tx=0x7f95f00042f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f95e7fff640 1 -- 172.21.15.73:0/3738761461 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0043020 con 0x7f96001003d0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96001a44e0 con 0x7f96001003d0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f95e7fff640 1 -- 172.21.15.73:0/3738761461 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f00054e0 con 0x7f96001003d0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f95e7fff640 1 -- 172.21.15.73:0/3738761461 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f0003b30 con 0x7f96001003d0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.032+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96001a4a80 con 0x7f96001003d0 2024-03-09T00:12:04.034 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.034+0000 7f95e7fff640 1 -- 172.21.15.73:0/3738761461 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f95f0046020 con 0x7f96001003d0 2024-03-09T00:12:04.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 Processor -- start 2024-03-09T00:12:04.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 -- start start 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea6bb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea6e460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95e7fff640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 0x7f95e80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fae9c1e00 con 0x564faea6c070 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fae96ab10 con 0x564faea5e490 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f9608009740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fae9bdea0 con 0x564faea5edb0 2024-03-09T00:12:04.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea6e460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 0x7f95e80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95feffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea6bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:46804/0 (socket says 172.21.15.73:46804) 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95feffd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea6bb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:44634/0 (socket says 172.21.15.73:44634) 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 -- 172.21.15.73:0/1900153735 learned_addr learned my addr 172.21.15.73:0/1900153735 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95e7fff640 1 -- 172.21.15.73:0/3738761461 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f95f00f1110 con 0x7f96001003d0 2024-03-09T00:12:04.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 -- 172.21.15.73:0/1900153735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 msgr2=0x564faea6bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea6bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 -- 172.21.15.73:0/1900153735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 msgr2=0x564faea6e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea6e460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95ff7fe640 1 -- 172.21.15.73:0/1900153735 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564fae7a07d0 con 0x564faea5e490 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.035+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 0x7f95e80a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9600068fa0 tx=0x7f95f4005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.036+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95f8012a20 tx=0x7f95f80096a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bc0fb6e3e7245800 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.036+0000 7f95e57fa640 1 -- 172.21.15.73:0/1900153735 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f8004b20 con 0x564faea5e490 2024-03-09T00:12:04.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.036+0000 7f95e57fa640 1 -- 172.21.15.73:0/1900153735 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f8004cc0 con 0x564faea5e490 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.036+0000 7f95e57fa640 1 -- 172.21.15.73:0/1900153735 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f80053f0 con 0x564faea5e490 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- 172.21.15.73:0/1900153735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 msgr2=0x564faea5e870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f95f8012a20 tx=0x7f95f80096a0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- 172.21.15.73:0/1900153735 shutdown_connections 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea6e460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea6bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- 172.21.15.73:0/1900153735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea5e870 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- 172.21.15.73:0/1900153735 >> 172.21.15.73:0/1900153735 conn(0x564faea58e90 msgr2=0x564faea59270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- 172.21.15.73:0/1900153735 shutdown_connections 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- 172.21.15.73:0/1900153735 wait complete. 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- start start 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 Processor -- start 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea6b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea68b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x564faea6ee30 con 0x564faea6c070 2024-03-09T00:12:04.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x564fae97e0a0 con 0x564faea5e490 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x564faea6efb0 con 0x564faea5edb0 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f9608009740 10 client.0 _ll_register_callbacks cb 0x564fae97da70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.037+0000 7f95feffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea6b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33386/0 (socket says 172.21.15.73:33386) 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 -- 172.21.15.73:0/4054352109 learned_addr learned my addr 172.21.15.73:0/4054352109 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:04.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea68b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 -- 172.21.15.73:0/4054352109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 msgr2=0x564faea68b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea68b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 -- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 msgr2=0x564faea6b7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea6b7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 -- 172.21.15.73:0/4054352109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564fae6d63f0 con 0x564faea6c070 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95feffd640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea6b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f95f80050d0 tx=0x7f95f8004bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20a5f76f5d97f7df server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f800fc30 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f95f8005170 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95f8004cc0 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95f8035050 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f9608009740 10 client.16563 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.038+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564fae6960f0 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f8004cc0 con 0x564faea6c070 2024-03-09T00:12:04.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f95f801f700 con 0x564faea6c070 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 10 client.16563 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 10 client.16563 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 20 client.16563 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faea7b9c0 0x564faea7ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 4 client.16563 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564fae9cf340 con 0x564faea7b9c0 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f95feffd640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faea7b9c0 0x564faea7ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f95feffd640 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faea7b9c0 0x564faea7ddb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=969984e11761f20f server_cookie=c4b83aec236b267e in_seq=0 out_seq=0 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.039+0000 7f95e2ff5640 10 client.16563 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f95e2ff5640 1 -- 172.21.15.73:0/4054352109 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x564fae9cf340 con 0x564faea7b9c0 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f95e2ff5640 10 client.16563 handle_command_reply: tid=0 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 client.16563 shutdown 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 msgr2=0x564faea69090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 crypto rx=0x7f95f80050d0 tx=0x7f95f8004bb0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 shutdown_connections 2024-03-09T00:12:04.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faea7b9c0 0x564faea7ddb0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea6c070 0x564faea69090 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea5edb0 0x564faea68b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f95e2ff5640 0 client.16563 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 --2- 172.21.15.73:0/4054352109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea6b7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 >> 172.21.15.73:0/4054352109 conn(0x564faea58e90 msgr2=0x564faea5abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 shutdown_connections 2024-03-09T00:12:04.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 1 -- 172.21.15.73:0/4054352109 wait complete. 2024-03-09T00:12:04.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.040+0000 7f9608009740 20 client.16563 trim_cache size 0 max 16384 2024-03-09T00:12:04.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 Processor -- start 2024-03-09T00:12:04.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 -- start start 2024-03-09T00:12:04.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea64360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.060 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea6b280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001aa500 con 0x564faea5edb0 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001acac0 con 0x564faea5e490 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f9608009740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001aab00 con 0x564faea6c070 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33394/0 (socket says 172.21.15.73:33394) 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 -- 172.21.15.73:0/2269899360 learned_addr learned my addr 172.21.15.73:0/2269899360 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:04.061 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95feffd640 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea64360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95ff7fe640 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea6b280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 -- 172.21.15.73:0/2269899360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 msgr2=0x564faea6b280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea6b280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 -- 172.21.15.73:0/2269899360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 msgr2=0x564faea64360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea64360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.057+0000 7f95fe7fc640 1 -- 172.21.15.73:0/2269899360 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564faea59c20 con 0x564faea5edb0 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.058+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f95f4004390 tx=0x7f95f4014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ebb69fd01695660b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.058+0000 7f95e4ff9640 1 -- 172.21.15.73:0/2269899360 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f4003a70 con 0x564faea5edb0 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.058+0000 7f95e4ff9640 1 -- 172.21.15.73:0/2269899360 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4003c10 con 0x564faea5edb0 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.058+0000 7f95e4ff9640 1 -- 172.21.15.73:0/2269899360 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f401c670 con 0x564faea5edb0 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- 172.21.15.73:0/2269899360 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 msgr2=0x564faea67b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 crypto rx=0x7f95f4004390 tx=0x7f95f4014000 comp rx=0 tx=0).stop 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- 172.21.15.73:0/2269899360 shutdown_connections 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea6b280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5edb0 0x564faea67b30 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- 172.21.15.73:0/2269899360 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5e490 0x564faea64360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- 172.21.15.73:0/2269899360 >> 172.21.15.73:0/2269899360 conn(0x564faea58e90 msgr2=0x564faea67420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- 172.21.15.73:0/2269899360 shutdown_connections 2024-03-09T00:12:04.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- 172.21.15.73:0/2269899360 wait complete. 2024-03-09T00:12:04.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- start start 2024-03-09T00:12:04.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 Processor -- start 2024-03-09T00:12:04.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5edb0 0x564faea6faf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea70230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001ab140 con 0x564faea5e490 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001ac460 con 0x564faea5edb0 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96001ab7a0 con 0x564faea6c070 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f9608009740 10 client.0 _ll_register_callbacks cb 0x7f95f8012510 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f95feffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f95fe7fc640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5edb0 0x564faea6faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f95feffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33396/0 (socket says 172.21.15.73:33396) 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f95ff7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea70230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.059+0000 7f95feffd640 1 -- 172.21.15.73:0/1470221443 learned_addr learned my addr 172.21.15.73:0/1470221443 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 -- 172.21.15.73:0/1470221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 msgr2=0x564faea70230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea70230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 -- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5edb0 msgr2=0x564faea6faf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5edb0 0x564faea6faf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 -- 172.21.15.73:0/1470221443 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564faead1e20 con 0x564faea5e490 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95feffd640 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f95f00ee690 tx=0x7f95f00a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a7c67fa012b27d89 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95f00ee750 con 0x564faea5e490 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f95f00b9d20 con 0x564faea5e490 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f95f4003c10 con 0x564faea5e490 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f95f002cb20 con 0x564faea5e490 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 10 client.16569 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:04.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x564faead3050 con 0x564faea5e490 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95f4003c10 con 0x564faea5e490 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f95f00d60b0 con 0x564faea5e490 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 10 client.16569 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 10 client.16569 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 20 client.16569 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faead36d0 0x564faead5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:04.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 4 client.16569 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.060+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564fae9c9690 con 0x564faead36d0 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faead36d0 0x564faead5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f95fe7fc640 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faead36d0 0x564faead5af0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=37664034566a28cd server_cookie=9eaf5783187a04d4 in_seq=0 out_seq=0 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f95e2ff5640 10 client.16569 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f95e2ff5640 1 -- 172.21.15.73:0/1470221443 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x564fae9c9690 con 0x564faead36d0 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f95e2ff5640 10 client.16569 handle_command_reply: tid=0 2024-03-09T00:12:04.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.061+0000 7f9608009740 1 client.16569 shutdown 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 msgr2=0x564faea73770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 secure :-1 s=READY pgs=802 cs=0 l=1 rev1=1 crypto rx=0x7f95f00ee690 tx=0x7f95f00a8040 comp rx=0 tx=0).stop 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 shutdown_connections 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x564faead36d0 0x564faead5af0 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x564faea6c070 0x564faea70230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x564faea5edb0 0x564faea6faf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f95e2ff5640 0 client.16569 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:04.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 --2- 172.21.15.73:0/1470221443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x564faea5e490 0x564faea73770 unknown :-1 s=CLOSED pgs=802 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 >> 172.21.15.73:0/1470221443 conn(0x564faea58e90 msgr2=0x564faea67cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 shutdown_connections 2024-03-09T00:12:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 1 -- 172.21.15.73:0/1470221443 wait complete. 2024-03-09T00:12:04.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9608009740 20 client.16569 trim_cache size 0 max 16384 2024-03-09T00:12:04.071 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2280 inodes in the stack)", 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "3da3df6c-042b-4abc-9499-61ed04ab646d": { 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "3da3df6c-042b-4abc-9499-61ed04ab646d", 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:12:04.072 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:12:04.073 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:04.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 msgr2=0x7f95e80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.074 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 0x7f95e80a4e30 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f9600068fa0 tx=0x7f95f4005f90 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 msgr2=0x7f9600198a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 crypto rx=0x7f95f002c240 tx=0x7f95f00042f0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 shutdown_connections 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f96001994d0 0x7f96001a3a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f96001054a0 0x7f9600198f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f95e80a2970 0x7f95e80a4e30 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 --2- 172.21.15.73:0/3738761461 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f96001003d0 0x7f9600198a00 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:04.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 >> 172.21.15.73:0/3738761461 conn(0x7f96000772f0 msgr2=0x7f96000fe3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:04.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 shutdown_connections 2024-03-09T00:12:04.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:04.062+0000 7f9605427640 1 -- 172.21.15.73:0/3738761461 wait complete. 2024-03-09T00:12:04.076 INFO:tasks.cephfs.filesystem:scrub status for tag:3da3df6c-042b-4abc-9499-61ed04ab646d - {'path': '/', 'tag': '3da3df6c-042b-4abc-9499-61ed04ab646d', 'options': 'recursive,force'} 2024-03-09T00:12:04.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:04 smithi073 ceph-mon[32309]: pgmap v159: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:12:04.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:04 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:12:04.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:04 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:04.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:04 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:12:04.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:04 smithi102 ceph-mon[36106]: pgmap v159: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:12:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:04 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:12:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:04 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:04 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:12:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:04 smithi179 ceph-mon[36637]: pgmap v159: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 22 KiB/s wr, 17 op/s 2024-03-09T00:12:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:04 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:12:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:04 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:04 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:12:05.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:05 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:05.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:05 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:05.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:05 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:06.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:06 smithi073 ceph-mon[32309]: pgmap v160: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:12:06.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:06 smithi102 ceph-mon[36106]: pgmap v160: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:12:06.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:06 smithi179 ceph-mon[36637]: pgmap v160: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 22 KiB/s wr, 16 op/s 2024-03-09T00:12:08.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:08 smithi073 ceph-mon[32309]: pgmap v161: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 22 KiB/s wr, 169 op/s 2024-03-09T00:12:08.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:08 smithi102 ceph-mon[36106]: pgmap v161: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 22 KiB/s wr, 169 op/s 2024-03-09T00:12:08.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:08 smithi179 ceph-mon[36637]: pgmap v161: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 163 KiB/s rd, 22 KiB/s wr, 169 op/s 2024-03-09T00:12:09.402 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:09 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:12:09.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:09 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:12:09.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:09 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:12:10.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:10 smithi073 ceph-mon[32309]: pgmap v162: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:10.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:10 smithi102 ceph-mon[36106]: pgmap v162: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:10.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:10 smithi179 ceph-mon[36637]: pgmap v162: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:12.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:12 smithi073 ceph-mon[32309]: pgmap v163: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:12.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:12 smithi102 ceph-mon[36106]: pgmap v163: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:12.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:12 smithi179 ceph-mon[36637]: pgmap v163: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 22 KiB/s wr, 303 op/s 2024-03-09T00:12:13.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:13 smithi102 ceph-mon[36106]: pgmap v164: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:12:13.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:13 smithi179 ceph-mon[36637]: pgmap v164: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:12:13.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:13 smithi073 ceph-mon[32309]: pgmap v164: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 22 KiB/s wr, 304 op/s 2024-03-09T00:12:16.214 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:15 smithi102 ceph-mon[36106]: pgmap v165: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-09T00:12:16.285 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:15 smithi179 ceph-mon[36637]: pgmap v165: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-09T00:12:16.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:15 smithi073 ceph-mon[32309]: pgmap v165: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 288 op/s 2024-03-09T00:12:18.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:18 smithi073 ceph-mon[32309]: pgmap v166: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 289 op/s 2024-03-09T00:12:18.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:18 smithi102 ceph-mon[36106]: pgmap v166: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 289 op/s 2024-03-09T00:12:18.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:18 smithi179 ceph-mon[36637]: pgmap v166: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 170 B/s wr, 289 op/s 2024-03-09T00:12:20.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:20 smithi073 ceph-mon[32309]: pgmap v167: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 85 B/s wr, 136 op/s 2024-03-09T00:12:20.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:20 smithi102 ceph-mon[36106]: pgmap v167: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 85 B/s wr, 136 op/s 2024-03-09T00:12:20.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:20 smithi179 ceph-mon[36637]: pgmap v167: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 85 B/s wr, 136 op/s 2024-03-09T00:12:22.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:22 smithi073 ceph-mon[32309]: pgmap v168: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:12:22.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:22 smithi102 ceph-mon[36106]: pgmap v168: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:12:22.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:22 smithi179 ceph-mon[36637]: pgmap v168: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 938 B/s rd, 85 B/s wr, 1 op/s 2024-03-09T00:12:24.407 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:24 smithi073 ceph-mon[32309]: pgmap v169: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:12:24.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:24 smithi102 ceph-mon[36106]: pgmap v169: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:12:24.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:24 smithi179 ceph-mon[36637]: pgmap v169: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:12:26.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:26 smithi073 ceph-mon[32309]: pgmap v170: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:26.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:26 smithi102 ceph-mon[36106]: pgmap v170: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:26.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:26 smithi179 ceph-mon[36637]: pgmap v170: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:28.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:28 smithi073 ceph-mon[32309]: pgmap v171: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:28.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:28 smithi102 ceph-mon[36106]: pgmap v171: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:28.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:28 smithi179 ceph-mon[36637]: pgmap v171: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:30.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:30 smithi073 ceph-mon[32309]: pgmap v172: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:30.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:30 smithi102 ceph-mon[36106]: pgmap v172: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:30.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:30 smithi179 ceph-mon[36637]: pgmap v172: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:32.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:32 smithi073 ceph-mon[32309]: pgmap v173: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:32.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:32 smithi102 ceph-mon[36106]: pgmap v173: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:32.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:32 smithi179 ceph-mon[36637]: pgmap v173: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:34.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:12:34.078 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:12:34.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- 172.21.15.73:0/2274255343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 msgr2=0x7f15981007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 --2- 172.21.15.73:0/2274255343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f15981007b0 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 crypto rx=0x7f158c009900 tx=0x7f158c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- 172.21.15.73:0/2274255343 shutdown_connections 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 --2- 172.21.15.73:0/2274255343 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 0x7f159810d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 --2- 172.21.15.73:0/2274255343 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598100d80 0x7f1598104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 --2- 172.21.15.73:0/2274255343 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f15981007b0 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- 172.21.15.73:0/2274255343 >> 172.21.15.73:0/2274255343 conn(0x7f15980772f0 msgr2=0x7f1598077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- 172.21.15.73:0/2274255343 shutdown_connections 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- 172.21.15.73:0/2274255343 wait complete. 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 Processor -- start 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.201+0000 7f159f092640 1 -- start start 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 0x7f159819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598196870 0x7f1598196d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1598110550 con 0x7f15981003d0 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981103d0 con 0x7f15981054a0 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981106d0 con 0x7f1598196870 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33668/0 (socket says 172.21.15.73:33668) 2024-03-09T00:12:34.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 -- 172.21.15.73:0/2565042247 learned_addr learned my addr 172.21.15.73:0/2565042247 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f1597fff640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 0x7f159819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159d62f640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598196870 0x7f1598196d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 -- 172.21.15.73:0/2565042247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598196870 msgr2=0x7f1598196d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598196870 0x7f1598196d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 -- 172.21.15.73:0/2565042247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 msgr2=0x7f159819cc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 0x7f159819cc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 -- 172.21.15.73:0/2565042247 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1598197390 con 0x7f15981003d0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f158c006010 tx=0x7f158c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f1595ffb640 1 -- 172.21.15.73:0/2565042247 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c041070 con 0x7f15981003d0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f1595ffb640 1 -- 172.21.15.73:0/2565042247 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f158c004450 con 0x7f15981003d0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f1595ffb640 1 -- 172.21.15.73:0/2565042247 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c034440 con 0x7f15981003d0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.202+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1598197620 con 0x7f15981003d0 2024-03-09T00:12:34.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.203+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15981a3810 con 0x7f15981003d0 2024-03-09T00:12:34.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.205+0000 7f1595ffb640 1 -- 172.21.15.73:0/2565042247 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f158c044070 con 0x7f15981003d0 2024-03-09T00:12:34.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 Processor -- start 2024-03-09T00:12:34.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 -- start start 2024-03-09T00:12:34.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb7596830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb75a4260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb74f9e00 con 0x559fb7596d70 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb74a2b10 con 0x559fb75a1e70 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f15a1c70740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb74f5ea0 con 0x559fb7596450 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f1595ffb640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 0x7f15880a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33672/0 (socket says 172.21.15.73:33672) 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 -- 172.21.15.73:0/2685148424 learned_addr learned my addr 172.21.15.73:0/2685148424 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f1597fff640 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb75a4260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159d62f640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 0x7f15880a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159d62f640 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb7596830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f1595ffb640 1 -- 172.21.15.73:0/2565042247 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f158c0f0cb0 con 0x7f15981003d0 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 -- 172.21.15.73:0/2685148424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 msgr2=0x559fb7596830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb7596830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 -- 172.21.15.73:0/2685148424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 msgr2=0x559fb75a4260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb75a4260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.206+0000 7f159ce2e640 1 -- 172.21.15.73:0/2685148424 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fb72d87d0 con 0x559fb7596d70 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f158c0bd300 tx=0x7f158c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94b63cc6bdc9ca9c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f159d62f640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 0x7f15880a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1590009fd0 tx=0x7f1590009840 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f1582ffd640 1 -- 172.21.15.73:0/2685148424 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c04b030 con 0x559fb7596d70 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f1582ffd640 1 -- 172.21.15.73:0/2685148424 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f158c0c11e0 con 0x559fb7596d70 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f1582ffd640 1 -- 172.21.15.73:0/2685148424 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f158c0035f0 con 0x559fb7596d70 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 -- 172.21.15.73:0/2685148424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 msgr2=0x559fb75a1930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 crypto rx=0x7f158c0bd300 tx=0x7f158c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 -- 172.21.15.73:0/2685148424 shutdown_connections 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb75a4260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a1930 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2685148424 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb7596830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 -- 172.21.15.73:0/2685148424 >> 172.21.15.73:0/2685148424 conn(0x559fb7590e90 msgr2=0x559fb7591270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 -- 172.21.15.73:0/2685148424 shutdown_connections 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.207+0000 7f15a1c70740 1 -- 172.21.15.73:0/2685148424 wait complete. 2024-03-09T00:12:34.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 -- start start 2024-03-09T00:12:34.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 Processor -- start 2024-03-09T00:12:34.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596450 0x559fb75a1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 0x559fb759ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb74f9e00 con 0x559fb7596d70 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb74b60a0 con 0x559fb7596450 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559fb75a4a20 con 0x559fb75a1e70 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f15a1c70740 10 client.0 _ll_register_callbacks cb 0x559fb74b5a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f1597fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f1597fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33686/0 (socket says 172.21.15.73:33686) 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f1597fff640 1 -- 172.21.15.73:0/3321717075 learned_addr learned my addr 172.21.15.73:0/3321717075 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.208+0000 7f159ce2e640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596450 0x559fb75a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f159d62f640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 0x559fb759ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 -- 172.21.15.73:0/3321717075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 msgr2=0x559fb759ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 0x559fb759ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 -- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596450 msgr2=0x559fb75a1590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596450 0x559fb75a1590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 -- 172.21.15.73:0/3321717075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fb720e3f0 con 0x559fb7596d70 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f159d62f640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 0x559fb759ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:34.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1597fff640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f158400b990 tx=0x7f158400be50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f15674be80bf0441 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1577fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1584005020 con 0x559fb7596d70 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f15840051c0 con 0x559fb7596d70 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f158c0c11e0 con 0x559fb7596d70 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f15840053c0 con 0x559fb7596d70 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f15a1c70740 10 client.16581 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559fb71ce0f0 con 0x559fb7596d70 2024-03-09T00:12:34.225 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.209+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f158c0c11e0 con 0x559fb7596d70 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f158401b630 con 0x559fb7596d70 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 10 client.16581 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 10 client.16581 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 20 client.16581 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb75b1610 0x559fb75b3a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 4 client.16581 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559fb7507340 con 0x559fb75b1610 2024-03-09T00:12:34.226 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f159ce2e640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb75b1610 0x559fb75b3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f1577fff640 10 client.16581 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.210+0000 7f159ce2e640 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb75b1610 0x559fb75b3a70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1172c33945132eac server_cookie=3ad81c0b18d2bd7c in_seq=0 out_seq=0 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f1577fff640 1 -- 172.21.15.73:0/3321717075 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559fb7507340 con 0x559fb75b1610 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f1577fff640 10 client.16581 handle_command_reply: tid=0 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 client.16581 shutdown 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 msgr2=0x559fb759e920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 crypto rx=0x7f158400b990 tx=0x7f158400be50 comp rx=0 tx=0).stop 2024-03-09T00:12:34.227 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 shutdown_connections 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb75b1610 0x559fb75b3a70 crc :-1 s=READY pgs=137 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb75a1e70 0x559fb759ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f1577fff640 0 client.16581 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb759e920 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3321717075 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596450 0x559fb75a1590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 >> 172.21.15.73:0/3321717075 conn(0x559fb7590e90 msgr2=0x559fb7592be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 shutdown_connections 2024-03-09T00:12:34.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 1 -- 172.21.15.73:0/3321717075 wait complete. 2024-03-09T00:12:34.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.211+0000 7f15a1c70740 20 client.16581 trim_cache size 0 max 16384 2024-03-09T00:12:34.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 Processor -- start 2024-03-09T00:12:34.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 -- start start 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75ade60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 0x559fb75a1050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981aa4c0 con 0x559fb75a1e70 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981aca80 con 0x559fb7596d70 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.226+0000 7f15a1c70740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981aaac0 con 0x559fb7596450 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33700/0 (socket says 172.21.15.73:33700) 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 -- 172.21.15.73:0/3546087727 learned_addr learned my addr 172.21.15.73:0/3546087727 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159ce2e640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75ade60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f1597fff640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 0x559fb75a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 -- 172.21.15.73:0/3546087727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 msgr2=0x559fb75ade60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75ade60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 -- 172.21.15.73:0/3546087727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 msgr2=0x559fb75a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 0x559fb75a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 -- 172.21.15.73:0/3546087727 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559fb75971b0 con 0x559fb75a1e70 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f1597fff640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 0x559fb75a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:34.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f159d62f640 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f15900124f0 tx=0x7f1590009600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34a8ef552b4e66e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f15827fc640 1 -- 172.21.15.73:0/3546087727 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590010ea0 con 0x559fb75a1e70 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f15827fc640 1 -- 172.21.15.73:0/3546087727 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f159002cb60 con 0x559fb75a1e70 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.227+0000 7f15827fc640 1 -- 172.21.15.73:0/3546087727 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1590021c00 con 0x559fb75a1e70 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- 172.21.15.73:0/3546087727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 msgr2=0x559fb75a1590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 crypto rx=0x7f15900124f0 tx=0x7f1590009600 comp rx=0 tx=0).stop 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- 172.21.15.73:0/3546087727 shutdown_connections 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb75a1e70 0x559fb75a1590 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb7596d70 0x559fb75a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 --2- 172.21.15.73:0/3546087727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75ade60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- 172.21.15.73:0/3546087727 >> 172.21.15.73:0/3546087727 conn(0x559fb7590e90 msgr2=0x559fb75ad750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- 172.21.15.73:0/3546087727 shutdown_connections 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- 172.21.15.73:0/3546087727 wait complete. 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 -- start start 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.228+0000 7f15a1c70740 1 Processor -- start 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75a5aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb7607b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981ab100 con 0x559fb7596d70 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981ac420 con 0x559fb75a1e70 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15981ab760 con 0x559fb7596450 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 10 client.0 _ll_register_callbacks cb 0x7f15840042b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:34.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33710/0 (socket says 172.21.15.73:33710) 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 -- 172.21.15.73:0/2582243379 learned_addr learned my addr 172.21.15.73:0/2582243379 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75a5aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f159d62f640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb7607b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 -- 172.21.15.73:0/2582243379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 msgr2=0x559fb75a5aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75a5aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 -- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 msgr2=0x559fb7607b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb7607b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 -- 172.21.15.73:0/2582243379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559fb76080d0 con 0x559fb7596d70 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f159d62f640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb7607b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1597fff640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f158401ba80 tx=0x7f1584004590 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c69f7385023ae38 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1577fff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:34.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f15840055d0 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f1584011650 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f159002cb60 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f158403b050 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 10 client.16587 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.229+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559fb76092a0 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f159002cb60 con 0x559fb7596d70 2024-03-09T00:12:34.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f15840329f0 con 0x559fb7596d70 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 10 client.16587 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 10 client.16587 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 20 client.16587 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb7609940 0x559fb760bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 4 client.16587 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x559fb7501690 con 0x559fb7609940 2024-03-09T00:12:34.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb7609940 0x559fb760bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f1577fff640 10 client.16587 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.230+0000 7f159ce2e640 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb7609940 0x559fb760bd60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7bb5daddd1b95e4 server_cookie=bd6e320aa578e7ae in_seq=0 out_seq=0 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f1577fff640 1 -- 172.21.15.73:0/2582243379 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x559fb7501690 con 0x559fb7609940 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f1577fff640 10 client.16587 handle_command_reply: tid=0 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 client.16587 shutdown 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 msgr2=0x559fb75a5fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 crypto rx=0x7f158401ba80 tx=0x7f1584004590 comp rx=0 tx=0).stop 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 shutdown_connections 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559fb7609940 0x559fb760bd60 crc :-1 s=READY pgs=138 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559fb75a1e70 0x559fb7607b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f1577fff640 0 client.16587 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559fb7596d70 0x559fb75a5fe0 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 --2- 172.21.15.73:0/2582243379 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559fb7596450 0x559fb75a5aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 >> 172.21.15.73:0/2582243379 conn(0x559fb7590e90 msgr2=0x559fb75998d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 shutdown_connections 2024-03-09T00:12:34.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 1 -- 172.21.15.73:0/2582243379 wait complete. 2024-03-09T00:12:34.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f15a1c70740 20 client.16587 trim_cache size 0 max 16384 2024-03-09T00:12:34.247 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:34.248 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:12:34.248 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:12:34.248 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:34.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 msgr2=0x7f15880a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 0x7f15880a4e30 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1590009fd0 tx=0x7f1590009840 comp rx=0 tx=0).stop 2024-03-09T00:12:34.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 msgr2=0x7f159819c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 crypto rx=0x7f158c006010 tx=0x7f158c02cc70 comp rx=0 tx=0).stop 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 shutdown_connections 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1598196870 0x7f1598196d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.231+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f15981054a0 0x7f159819cc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.232+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f15880a2970 0x7f15880a4e30 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.232+0000 7f159f092640 1 --2- 172.21.15.73:0/2565042247 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f15981003d0 0x7f159819c6c0 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.232+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 >> 172.21.15.73:0/2565042247 conn(0x7f15980772f0 msgr2=0x7f15980fe300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.232+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 shutdown_connections 2024-03-09T00:12:34.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.232+0000 7f159f092640 1 -- 172.21.15.73:0/2565042247 wait complete. 2024-03-09T00:12:34.252 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:12:34.253 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:12:34.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.357+0000 7f60d59e6640 1 -- 172.21.15.73:0/2433196437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 msgr2=0x7f60d010e2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.357+0000 7f60d59e6640 1 --2- 172.21.15.73:0/2433196437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d010e2c0 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 crypto rx=0x7f60c8009ca0 tx=0x7f60c802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- 172.21.15.73:0/2433196437 shutdown_connections 2024-03-09T00:12:34.357 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- 172.21.15.73:0/2433196437 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d010e2c0 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- 172.21.15.73:0/2433196437 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d01008a0 0x7f60d0105fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- 172.21.15.73:0/2433196437 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d00ffef0 0x7f60d01002d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- 172.21.15.73:0/2433196437 >> 172.21.15.73:0/2433196437 conn(0x7f60d0075500 msgr2=0x7f60d0075910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- 172.21.15.73:0/2433196437 shutdown_connections 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- 172.21.15.73:0/2433196437 wait complete. 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 Processor -- start 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- start start 2024-03-09T00:12:34.358 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d00ffef0 0x7f60d0100e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d01018e0 0x7f60d01a5e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d01117a0 con 0x7f60d01066b0 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d0111620 con 0x7f60d00ffef0 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.358+0000 7f60d59e6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d0111920 con 0x7f60d01018e0 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33724/0 (socket says 172.21.15.73:33724) 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 -- 172.21.15.73:0/1023009464 learned_addr learned my addr 172.21.15.73:0/1023009464 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.359 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60ceffd640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d00ffef0 0x7f60d0100e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 -- 172.21.15.73:0/1023009464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d01018e0 msgr2=0x7f60d01a5e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60cf7fe640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d01018e0 0x7f60d01a5e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d01018e0 0x7f60d01a5e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 -- 172.21.15.73:0/1023009464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d00ffef0 msgr2=0x7f60d0100e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d00ffef0 0x7f60d0100e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 -- 172.21.15.73:0/1023009464 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d01a6510 con 0x7f60d01066b0 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60c67fc640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f60b800d770 tx=0x7f60b800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b8017070 con 0x7f60d01066b0 2024-03-09T00:12:34.360 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d01a6800 con 0x7f60d01066b0 2024-03-09T00:12:34.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60b8009c30 con 0x7f60d01066b0 2024-03-09T00:12:34.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d01a6da0 con 0x7f60d01066b0 2024-03-09T00:12:34.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.359+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60b8021060 con 0x7f60d01066b0 2024-03-09T00:12:34.361 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.360+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60d0069420 con 0x7f60d01066b0 2024-03-09T00:12:34.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.361+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f60b801a070 con 0x7f60d01066b0 2024-03-09T00:12:34.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.362+0000 7f60d49e4640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 0x7f60bc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.362+0000 7f60ceffd640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 0x7f60bc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.362+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f60b80cf030 con 0x7f60d01066b0 2024-03-09T00:12:34.362 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.362+0000 7f60ceffd640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 0x7f60bc0a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f60c0005fd0 tx=0x7f60c000b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.363 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.365+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f60b8009de0 con 0x7f60d01066b0 2024-03-09T00:12:34.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:34 smithi073 ceph-mon[32309]: pgmap v174: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:34.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:34 smithi102 ceph-mon[36106]: pgmap v174: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:34.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.576+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f60d0109980 con 0x7f60d01066b0 2024-03-09T00:12:34.573 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.577+0000 7f60d49e4640 1 -- 172.21.15.73:0/1023009464 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f60b809b3b0 con 0x7f60d01066b0 2024-03-09T00:12:34.574 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:12:34.574 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:12:34.574 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 msgr2=0x7f60bc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 0x7f60bc0a4e30 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f60c0005fd0 tx=0x7f60c000b390 comp rx=0 tx=0).stop 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 msgr2=0x7f60d01013a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 crypto rx=0x7f60b800d770 tx=0x7f60b800dc30 comp rx=0 tx=0).stop 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 shutdown_connections 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f60d01018e0 0x7f60d01a5e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.576 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60bc0a2970 0x7f60bc0a4e30 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f60d01066b0 0x7f60d01013a0 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 --2- 172.21.15.73:0/1023009464 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f60d00ffef0 0x7f60d0100e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 >> 172.21.15.73:0/1023009464 conn(0x7f60d0075500 msgr2=0x7f60d00fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 shutdown_connections 2024-03-09T00:12:34.577 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.579+0000 7f60d59e6640 1 -- 172.21.15.73:0/1023009464 wait complete. 2024-03-09T00:12:34.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:34 smithi179 ceph-mon[36637]: pgmap v174: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:34.587 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:12:34.587 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:12:34.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 -- 172.21.15.73:0/2184563559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 msgr2=0x7fa0b8069e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 --2- 172.21.15.73:0/2184563559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b8069e20 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009930 tx=0x7fa0ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:34.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 -- 172.21.15.73:0/2184563559 shutdown_connections 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 --2- 172.21.15.73:0/2184563559 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 0x7fa0b810e130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 --2- 172.21.15.73:0/2184563559 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b8069e20 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 --2- 172.21.15.73:0/2184563559 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 0x7fa0b8107790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 -- 172.21.15.73:0/2184563559 >> 172.21.15.73:0/2184563559 conn(0x7fa0b8076f90 msgr2=0x7fa0b80773a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 -- 172.21.15.73:0/2184563559 shutdown_connections 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.694+0000 7fa0bda73640 1 -- 172.21.15.73:0/2184563559 wait complete. 2024-03-09T00:12:34.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 Processor -- start 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 -- start start 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 0x7fa0b819dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 0x7fa0b81978c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b8111640 con 0x7fa0b80699a0 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81114c0 con 0x7fa0b81073b0 2024-03-09T00:12:34.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0bda73640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81117c0 con 0x7fa0b806a360 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33746/0 (socket says 172.21.15.73:33746) 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1164075590 learned_addr learned my addr 172.21.15.73:0/1164075590 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 0x7fa0b81978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.695+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 0x7fa0b819dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1164075590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 msgr2=0x7fa0b819dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 0x7fa0b819dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1164075590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 msgr2=0x7fa0b81978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 0x7fa0b81978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1164075590 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0b8198040 con 0x7fa0b80699a0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800e9f0 tx=0x7fa0a800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b4ff9640 1 -- 172.21.15.73:0/1164075590 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a800cd80 con 0x7fa0b80699a0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b4ff9640 1 -- 172.21.15.73:0/1164075590 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8010910 con 0x7fa0b80699a0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b8198330 con 0x7fa0b80699a0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0b4ff9640 1 -- 172.21.15.73:0/1164075590 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a8018990 con 0x7fa0b80699a0 2024-03-09T00:12:34.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.696+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b81a45d0 con 0x7fa0b80699a0 2024-03-09T00:12:34.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b4ff9640 1 -- 172.21.15.73:0/1164075590 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa0a8010ac0 con 0x7fa0b80699a0 2024-03-09T00:12:34.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 Processor -- start 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 -- start start 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16d0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16d2a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc168e360 con 0x558fc16c71a0 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc162aec0 con 0x558fc16c7ac0 2024-03-09T00:12:34.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0c0653740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc15d3d30 con 0x558fc16d06a0 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b4ff9640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 0x7fa0a00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33752/0 (socket says 172.21.15.73:33752) 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 -- 172.21.15.73:0/946808995 learned_addr learned my addr 172.21.15.73:0/946808995 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 0x7fa0a00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16d2a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b4ff9640 1 -- 172.21.15.73:0/1164075590 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa0a80cf550 con 0x7fa0b80699a0 2024-03-09T00:12:34.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16d0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 -- 172.21.15.73:0/946808995 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 msgr2=0x558fc16d2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16d2a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 -- 172.21.15.73:0/946808995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 msgr2=0x558fc16d0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.699+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16d0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa0b7fff640 1 -- 172.21.15.73:0/946808995 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fc15d2bc0 con 0x558fc16c71a0 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0011e20 tx=0x7fa0b000f790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ea234b35bdb3e044 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 0x7fa0a00a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000fc90 tx=0x7fa0b001e000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa09dffb640 1 -- 172.21.15.73:0/946808995 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b001dcb0 con 0x558fc16c71a0 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa09dffb640 1 -- 172.21.15.73:0/946808995 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0b001de50 con 0x558fc16c71a0 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.700+0000 7fa09dffb640 1 -- 172.21.15.73:0/946808995 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0b0004290 con 0x558fc16c71a0 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- 172.21.15.73:0/946808995 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 msgr2=0x558fc16c7580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 crypto rx=0x7fa0b0011e20 tx=0x7fa0b000f790 comp rx=0 tx=0).stop 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- 172.21.15.73:0/946808995 shutdown_connections 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16d2a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16d0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- 172.21.15.73:0/946808995 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16c7580 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- 172.21.15.73:0/946808995 >> 172.21.15.73:0/946808995 conn(0x558fc16c18d0 msgr2=0x558fc16c1cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- 172.21.15.73:0/946808995 shutdown_connections 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- 172.21.15.73:0/946808995 wait complete. 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- start start 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 Processor -- start 2024-03-09T00:12:34.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c71a0 0x558fc16cd6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16ce150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc15c94a0 con 0x558fc16c7ac0 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc1626f60 con 0x558fc16c71a0 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558fc16d33d0 con 0x558fc16d06a0 2024-03-09T00:12:34.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.701+0000 7fa0c0653740 10 client.0 _ll_register_callbacks cb 0x558fc1624d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33768/0 (socket says 172.21.15.73:33768) 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2556576434 learned_addr learned my addr 172.21.15.73:0/2556576434 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2556576434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 msgr2=0x558fc16ce150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16ce150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.712 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c71a0 0x558fc16cd6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c71a0 msgr2=0x558fc16cd6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c71a0 0x558fc16cd6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2556576434 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fc133f3f0 con 0x558fc16c7ac0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009900 tx=0x7fa0ac02cdf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f99fa8aedb7c21c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac004460 con 0x558fc16c7ac0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa0ac0046c0 con 0x558fc16c7ac0 2024-03-09T00:12:34.713 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0b001de50 con 0x558fc16c7ac0 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0ac050050 con 0x558fc16c7ac0 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0c0653740 10 client.16605 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.702+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558fc14097d0 con 0x558fc16c7ac0 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0b001de50 con 0x558fc16c7ac0 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa0ac0035f0 con 0x558fc16c7ac0 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 10 client.16605 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:34.714 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 10 client.16605 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:34.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 20 client.16605 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:34.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc16e0750 0x558fc16e2b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 4 client.16605 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:34.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558fc1638400 con 0x558fc16e0750 2024-03-09T00:12:34.715 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.703+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc16e0750 0x558fc16e2b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa096ffd640 10 client.16605 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc16e0750 0x558fc16e2b80 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=93d4da55a7434065 server_cookie=d3e5fb95703ba38f in_seq=0 out_seq=0 2024-03-09T00:12:34.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa096ffd640 1 -- 172.21.15.73:0/2556576434 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558fc1638400 con 0x558fc16e0750 2024-03-09T00:12:34.716 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa096ffd640 10 client.16605 handle_command_reply: tid=0 2024-03-09T00:12:34.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 client.16605 shutdown 2024-03-09T00:12:34.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 msgr2=0x558fc16cdc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac009900 tx=0x7fa0ac02cdf0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 shutdown_connections 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc16e0750 0x558fc16e2b80 crc :-1 s=READY pgs=139 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16ce150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16cdc10 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa096ffd640 0 client.16605 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2556576434 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c71a0 0x558fc16cd6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 >> 172.21.15.73:0/2556576434 conn(0x558fc16c18d0 msgr2=0x558fc16c3620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 shutdown_connections 2024-03-09T00:12:34.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 1 -- 172.21.15.73:0/2556576434 wait complete. 2024-03-09T00:12:34.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.704+0000 7fa0c0653740 20 client.16605 trim_cache size 0 max 16384 2024-03-09T00:12:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 Processor -- start 2024-03-09T00:12:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 -- start start 2024-03-09T00:12:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16cbfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16cc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ac6d0 con 0x558fc16c71a0 2024-03-09T00:12:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81aec90 con 0x558fc16c7ac0 2024-03-09T00:12:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0c0653740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81accd0 con 0x558fc16d06a0 2024-03-09T00:12:34.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0b77fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33780/0 (socket says 172.21.15.73:33780) 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1802945336 learned_addr learned my addr 172.21.15.73:0/1802945336 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.716+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16cbfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16cc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1802945336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 msgr2=0x558fc16cc4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16cc4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1802945336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 msgr2=0x558fc16cbfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16cbfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 -- 172.21.15.73:0/1802945336 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558fc16e7dc0 con 0x558fc16c71a0 2024-03-09T00:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80ccf10 tx=0x7fa0a806d5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c4d64e55100b7df7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.732 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa09d7fa640 1 -- 172.21.15.73:0/1802945336 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a80ccb90 con 0x558fc16c71a0 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa09d7fa640 1 -- 172.21.15.73:0/1802945336 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8097f40 con 0x558fc16c71a0 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.717+0000 7fa09d7fa640 1 -- 172.21.15.73:0/1802945336 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0a809cb50 con 0x558fc16c71a0 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- 172.21.15.73:0/1802945336 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 msgr2=0x558fc16cfe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 crypto rx=0x7fa0a80ccf10 tx=0x7fa0a806d5c0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- 172.21.15.73:0/1802945336 shutdown_connections 2024-03-09T00:12:34.733 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16d06a0 0x558fc16cc4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16c7ac0 0x558fc16cbfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 --2- 172.21.15.73:0/1802945336 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c71a0 0x558fc16cfe00 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- 172.21.15.73:0/1802945336 >> 172.21.15.73:0/1802945336 conn(0x558fc16c18d0 msgr2=0x558fc16c89e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- 172.21.15.73:0/1802945336 shutdown_connections 2024-03-09T00:12:34.734 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- 172.21.15.73:0/1802945336 wait complete. 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.718+0000 7fa0c0653740 1 -- start start 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 Processor -- start 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16c71a0 0x558fc16d7af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 0x558fc16d65d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.735 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ad310 con 0x558fc16c7ac0 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ae630 con 0x558fc16d06a0 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b81ad970 con 0x558fc16c71a0 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0c0653740 10 client.0 _ll_register_callbacks cb 0x7fa0b0011460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33792/0 (socket says 172.21.15.73:33792) 2024-03-09T00:12:34.736 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2666093388 learned_addr learned my addr 172.21.15.73:0/2666093388 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16c71a0 0x558fc16d7af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 0x558fc16d65d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2666093388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16c71a0 msgr2=0x558fc16d7af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16c71a0 0x558fc16d7af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 msgr2=0x558fc16d65d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 0x558fc16d65d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.737 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 -- 172.21.15.73:0/2666093388 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558fc1736cc0 con 0x558fc16c7ac0 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b7fff640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 0x558fc16d65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa0b6ffd640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac006010 tx=0x7fa0ac02c330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=532d2d96a9648ceb server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa096ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa0ac003800 con 0x558fc16c7ac0 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.719+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa0ac004460 con 0x558fc16c7ac0 2024-03-09T00:12:34.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa0a8097f40 con 0x558fc16c7ac0 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa0ac02cc20 con 0x558fc16c7ac0 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 10 client.16611 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558fc1737eb0 con 0x558fc16c7ac0 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa0a8097f40 con 0x558fc16c7ac0 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa0ac005300 con 0x558fc16c7ac0 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 10 client.16611 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 10 client.16611 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:34.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 20 client.16611 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc1738580 0x558fc173a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 4 client.16611 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x558fc1632750 con 0x558fc1738580 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.720+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc1738580 0x558fc173a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0b77fe640 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc1738580 0x558fc173a9a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad851ab245258ca1 server_cookie=ce8c2196e9b55f01 in_seq=0 out_seq=0 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa096ffd640 10 client.16611 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa096ffd640 1 -- 172.21.15.73:0/2666093388 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x558fc1632750 con 0x558fc1738580 2024-03-09T00:12:34.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa096ffd640 10 client.16611 handle_command_reply: tid=0 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 client.16611 shutdown 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 msgr2=0x558fc16d5e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 crypto rx=0x7fa0ac006010 tx=0x7fa0ac02c330 comp rx=0 tx=0).stop 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 shutdown_connections 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558fc1738580 0x558fc173a9a0 crc :-1 s=READY pgs=140 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558fc16d06a0 0x558fc16d65d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558fc16c7ac0 0x558fc16d5e90 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa096ffd640 0 client.16611 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 --2- 172.21.15.73:0/2666093388 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558fc16c71a0 0x558fc16d7af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 >> 172.21.15.73:0/2666093388 conn(0x558fc16c18d0 msgr2=0x558fc16cbf40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 shutdown_connections 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 1 -- 172.21.15.73:0/2666093388 wait complete. 2024-03-09T00:12:34.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.721+0000 7fa0c0653740 20 client.16611 trim_cache size 0 max 16384 2024-03-09T00:12:34.743 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:12:34.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 msgr2=0x7fa0a00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 0x7fa0a00a4e30 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7fa0b000fc90 tx=0x7fa0b001e000 comp rx=0 tx=0).stop 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 msgr2=0x7fa0b819d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 crypto rx=0x7fa0a800e9f0 tx=0x7fa0a800eeb0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 shutdown_connections 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa0b81073b0 0x7fa0b81978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa0b806a360 0x7fa0b819dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa0a00a2970 0x7fa0a00a4e30 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.745 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 --2- 172.21.15.73:0/1164075590 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa0b80699a0 0x7fa0b819d710 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:34.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 >> 172.21.15.73:0/1164075590 conn(0x7fa0b8076f90 msgr2=0x7fa0b80fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:34.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 shutdown_connections 2024-03-09T00:12:34.746 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:34.722+0000 7fa0bda73640 1 -- 172.21.15.73:0/1164075590 wait complete. 2024-03-09T00:12:35.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:35 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1023009464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:35.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:35 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1023009464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:35.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:35 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1023009464' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:12:35.747 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:12:35.747 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:12:35.748 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:12:35.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 -- 172.21.15.73:0/3635824363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c102580 msgr2=0x7fd74c102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 --2- 172.21.15.73:0/3635824363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c102580 0x7fd74c102960 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 crypto rx=0x7fd73c009900 tx=0x7fd73c02bcd0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 -- 172.21.15.73:0/3635824363 shutdown_connections 2024-03-09T00:12:35.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 --2- 172.21.15.73:0/3635824363 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c10b950 0x7fd74c10dd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 --2- 172.21.15.73:0/3635824363 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 0x7fd74c10b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 --2- 172.21.15.73:0/3635824363 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c102580 0x7fd74c102960 unknown :-1 s=CLOSED pgs=817 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 -- 172.21.15.73:0/3635824363 >> 172.21.15.73:0/3635824363 conn(0x7fd74c077350 msgr2=0x7fd74c077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 -- 172.21.15.73:0/3635824363 shutdown_connections 2024-03-09T00:12:35.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.864+0000 7fd751fab640 1 -- 172.21.15.73:0/3635824363 wait complete. 2024-03-09T00:12:35.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 Processor -- start 2024-03-09T00:12:35.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 -- start start 2024-03-09T00:12:35.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c102580 0x7fd74c19d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 0x7fd74c19ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1116d0 con 0x7fd74c10b950 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c111550 con 0x7fd74c102580 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd751fab640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c111850 con 0x7fd74c102ea0 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33820/0 (socket says 172.21.15.73:33820) 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd74bfff640 1 -- 172.21.15.73:0/1965133546 learned_addr learned my addr 172.21.15.73:0/1965133546 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd74affd640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 0x7fd74c19ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.865+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c102580 0x7fd74c19d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 -- 172.21.15.73:0/1965133546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 msgr2=0x7fd74c19ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 0x7fd74c19ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 -- 172.21.15.73:0/1965133546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c102580 msgr2=0x7fd74c19d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c102580 0x7fd74c19d880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 -- 172.21.15.73:0/1965133546 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd74c1981b0 con 0x7fd74c10b950 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd74bfff640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd74400eaa0 tx=0x7fd74400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd748ff9640 1 -- 172.21.15.73:0/1965133546 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74400cdb0 con 0x7fd74c10b950 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd748ff9640 1 -- 172.21.15.73:0/1965133546 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd744005020 con 0x7fd74c10b950 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd74c1984a0 con 0x7fd74c10b950 2024-03-09T00:12:35.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd748ff9640 1 -- 172.21.15.73:0/1965133546 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440116b0 con 0x7fd74c10b950 2024-03-09T00:12:35.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.866+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd74c1a4740 con 0x7fd74c10b950 2024-03-09T00:12:35.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.868+0000 7fd748ff9640 1 -- 172.21.15.73:0/1965133546 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd744005560 con 0x7fd74c10b950 2024-03-09T00:12:35.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 Processor -- start 2024-03-09T00:12:35.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 -- start start 2024-03-09T00:12:35.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79befd80 0x562b79bf0160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd748ff9640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 0x7fd7340a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79bf9280 0x562b79bfb670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79b53b40 con 0x562b79bf06a0 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79afcd80 con 0x562b79befd80 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd754b82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79b4fbe0 con 0x562b79bf9280 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 0x7fd7340a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33836/0 (socket says 172.21.15.73:33836) 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 -- 172.21.15.73:0/1567410592 learned_addr learned my addr 172.21.15.73:0/1567410592 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd748ff9640 1 -- 172.21.15.73:0/1965133546 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd7440cf450 con 0x7fd74c10b950 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79befd80 0x562b79bf0160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74bfff640 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79bf9280 0x562b79bfb670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 -- 172.21.15.73:0/1567410592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79bf9280 msgr2=0x562b79bfb670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79bf9280 0x562b79bfb670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 -- 172.21.15.73:0/1567410592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79befd80 msgr2=0x562b79bf0160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79befd80 0x562b79bf0160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74affd640 1 -- 172.21.15.73:0/1567410592 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b799307e0 con 0x562b79bf06a0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.869+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 0x7fd7340a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd73c006010 tx=0x7fd73c009610 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd74affd640 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd740011700 tx=0x7fd74000a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7cf437ce1728959c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd731ffb640 1 -- 172.21.15.73:0/1567410592 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740004020 con 0x562b79bf06a0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd731ffb640 1 -- 172.21.15.73:0/1567410592 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd740005500 con 0x562b79bf06a0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd731ffb640 1 -- 172.21.15.73:0/1567410592 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd740004b60 con 0x562b79bf06a0 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 -- 172.21.15.73:0/1567410592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 msgr2=0x562b79bf8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 crypto rx=0x7fd740011700 tx=0x7fd74000a6a0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 -- 172.21.15.73:0/1567410592 shutdown_connections 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79bf9280 0x562b79bfb670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf8d40 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 --2- 172.21.15.73:0/1567410592 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79befd80 0x562b79bf0160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 -- 172.21.15.73:0/1567410592 >> 172.21.15.73:0/1567410592 conn(0x562b79bea4b0 msgr2=0x562b79bea890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 -- 172.21.15.73:0/1567410592 shutdown_connections 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.870+0000 7fd754b82740 1 -- 172.21.15.73:0/1567410592 wait complete. 2024-03-09T00:12:35.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 -- start start 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 Processor -- start 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf9280 0x562b79bf6630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79bfc070 con 0x562b79bf06a0 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79b101c0 con 0x562b79bf9280 2024-03-09T00:12:35.882 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x562b79bfc1f0 con 0x562b79befd80 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd754b82740 10 client.0 _ll_register_callbacks cb 0x562b79aff7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd74affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd74affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33840/0 (socket says 172.21.15.73:33840) 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd74affd640 1 -- 172.21.15.73:0/1411049811 learned_addr learned my addr 172.21.15.73:0/1411049811 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.871+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74bfff640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf9280 0x562b79bf6630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 -- 172.21.15.73:0/1411049811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 msgr2=0x562b79bf8c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf8c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.883 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 -- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf9280 msgr2=0x562b79bf6630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf9280 0x562b79bf6630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 -- 172.21.15.73:0/1411049811 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b79990f70 con 0x562b79bf06a0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd74affd640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd740004b50 tx=0x7fd740005490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=440336da4e00b560 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74002c060 con 0x562b79bf06a0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd74000f990 con 0x562b79bf06a0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd740005500 con 0x562b79bf06a0 2024-03-09T00:12:35.884 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd740033050 con 0x562b79bf06a0 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 10 client.16623 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562b79828170 con 0x562b79bf06a0 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd740005500 con 0x562b79bf06a0 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd740026030 con 0x562b79bf06a0 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 10 client.16623 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 10 client.16623 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 20 client.16623 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.872+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c09280 0x562b79c0b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.873+0000 7fd754b82740 4 client.16623 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.873+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562b79b29960 con 0x562b79c09280 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.873+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c09280 0x562b79c0b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.873+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c09280 0x562b79c0b670 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ba35957bbefe760a server_cookie=9df2c9afe565ca6f in_seq=0 out_seq=0 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.873+0000 7fd72affd640 10 client.16623 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd72affd640 1 -- 172.21.15.73:0/1411049811 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x562b79b29960 con 0x562b79c09280 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd72affd640 10 client.16623 handle_command_reply: tid=0 2024-03-09T00:12:35.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 client.16623 shutdown 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 msgr2=0x562b79bf6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 crypto rx=0x7fd740004b50 tx=0x7fd740005490 comp rx=0 tx=0).stop 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 shutdown_connections 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c09280 0x562b79c0b670 crc :-1 s=READY pgs=141 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf9280 0x562b79bf6630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf06a0 0x562b79bf6020 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd72affd640 0 client.16623 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:35.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 --2- 172.21.15.73:0/1411049811 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf8c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 >> 172.21.15.73:0/1411049811 conn(0x562b79bea4b0 msgr2=0x562b79bec200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 shutdown_connections 2024-03-09T00:12:35.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 1 -- 172.21.15.73:0/1411049811 wait complete. 2024-03-09T00:12:35.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.874+0000 7fd754b82740 20 client.16623 trim_cache size 0 max 16384 2024-03-09T00:12:35.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 Processor -- start 2024-03-09T00:12:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 -- start start 2024-03-09T00:12:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bf8750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1ac760 con 0x562b79bf9280 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1aed20 con 0x562b79bf06a0 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd754b82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1acd60 con 0x562b79befd80 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33852/0 (socket says 172.21.15.73:33852) 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 -- 172.21.15.73:0/3211392644 learned_addr learned my addr 172.21.15.73:0/3211392644 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74affd640 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bf8750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 -- 172.21.15.73:0/3211392644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 msgr2=0x562b79bf5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 -- 172.21.15.73:0/3211392644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 msgr2=0x562b79bf8750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bf8750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 -- 172.21.15.73:0/3211392644 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562b79bf69a0 con 0x562b79bf9280 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd74bfff640 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd74409b3c0 tx=0x7fd74409b5c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=93d51fabcad28ee7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd7317fa640 1 -- 172.21.15.73:0/3211392644 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd74401f040 con 0x562b79bf9280 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd7317fa640 1 -- 172.21.15.73:0/3211392644 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7440ade90 con 0x562b79bf9280 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.890+0000 7fd7317fa640 1 -- 172.21.15.73:0/3211392644 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd744097d30 con 0x562b79bf9280 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- 172.21.15.73:0/3211392644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 msgr2=0x562b79bf8c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 crypto rx=0x7fd74409b3c0 tx=0x7fd74409b5c0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- 172.21.15.73:0/3211392644 shutdown_connections 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bf8c90 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bf8750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 --2- 172.21.15.73:0/3211392644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79bf5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- 172.21.15.73:0/3211392644 >> 172.21.15.73:0/3211392644 conn(0x562b79bea4b0 msgr2=0x562b79bf1320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- 172.21.15.73:0/3211392644 shutdown_connections 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- 172.21.15.73:0/3211392644 wait complete. 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.891+0000 7fd754b82740 1 -- start start 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 Processor -- start 2024-03-09T00:12:35.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79c00680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bfe470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1ad3a0 con 0x562b79bf9280 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1ae6c0 con 0x562b79bf06a0 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd74c1ada00 con 0x562b79befd80 2024-03-09T00:12:35.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 10 client.0 _ll_register_callbacks cb 0x7fd74000a9f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33858/0 (socket says 172.21.15.73:33858) 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 -- 172.21.15.73:0/4139283181 learned_addr learned my addr 172.21.15.73:0/4139283181 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74affd640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bfe470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79c00680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 -- 172.21.15.73:0/4139283181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 msgr2=0x562b79c00680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79c00680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 -- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 msgr2=0x562b79bfe470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bfe470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 -- 172.21.15.73:0/4139283181 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562b79c0b9e0 con 0x562b79bf9280 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79c00680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:12:35.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd74bfff640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd7440c7620 tx=0x7fd7440058d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=adfc07bc42d22fff server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd72affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7440cb070 con 0x562b79bf9280 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fd7440cf040 con 0x562b79bf9280 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.892+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7440ade90 con 0x562b79bf9280 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd74401f040 con 0x562b79bf9280 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 10 client.16629 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:35.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x562b79c61390 con 0x562b79bf9280 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7440ade90 con 0x562b79bf9280 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd74400f040 con 0x562b79bf9280 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 10 client.16629 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 10 client.16629 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 20 client.16629 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c61a30 0x562b79c63e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:35.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 4 client.16629 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:12:35.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x562b79b15080 con 0x562b79c61a30 2024-03-09T00:12:35.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.893+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c61a30 0x562b79c63e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:35.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.894+0000 7fd72affd640 10 client.16629 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:35.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.894+0000 7fd74b7fe640 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c61a30 0x562b79c63e50 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3111db02523e9ac server_cookie=b8fad0cfd15d970f in_seq=0 out_seq=0 2024-03-09T00:12:35.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd72affd640 1 -- 172.21.15.73:0/4139283181 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x562b79b15080 con 0x562b79c61a30 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd72affd640 10 client.16629 handle_command_reply: tid=0 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 client.16629 shutdown 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 msgr2=0x562b79bfe9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 crypto rx=0x7fd7440c7620 tx=0x7fd7440058d0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 shutdown_connections 2024-03-09T00:12:35.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x562b79c61a30 0x562b79c63e50 crc :-1 s=READY pgs=142 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x562b79bf9280 0x562b79bfe9b0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x562b79bf06a0 0x562b79bfe470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 --2- 172.21.15.73:0/4139283181 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x562b79befd80 0x562b79c00680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 >> 172.21.15.73:0/4139283181 conn(0x562b79bea4b0 msgr2=0x562b79be8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd72affd640 0 client.16629 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:35.908 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 shutdown_connections 2024-03-09T00:12:35.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 1 -- 172.21.15.73:0/4139283181 wait complete. 2024-03-09T00:12:35.909 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.895+0000 7fd754b82740 20 client.16629 trim_cache size 0 max 16384 2024-03-09T00:12:35.909 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:35.910 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:12:35.910 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "31d89c64-3ce3-4b02-ad1c-eb94219b9378", 2024-03-09T00:12:35.910 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:12:35.910 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:35.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 msgr2=0x7fd7340a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 0x7fd7340a4e30 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fd73c006010 tx=0x7fd73c009610 comp rx=0 tx=0).stop 2024-03-09T00:12:35.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 msgr2=0x7fd74c197a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:35.912 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 crypto rx=0x7fd74400eaa0 tx=0x7fd74400ef60 comp rx=0 tx=0).stop 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 shutdown_connections 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd7340a2970 0x7fd7340a4e30 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd74c10b950 0x7fd74c197a30 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd74c102ea0 0x7fd74c19ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 --2- 172.21.15.73:0/1965133546 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd74c102580 0x7fd74c19d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 >> 172.21.15.73:0/1965133546 conn(0x7fd74c077350 msgr2=0x7fd74c0fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 shutdown_connections 2024-03-09T00:12:35.913 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:35.896+0000 7fd751fab640 1 -- 172.21.15.73:0/1965133546 wait complete. 2024-03-09T00:12:35.914 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:12:35.914 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:12:36.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 -- 172.21.15.73:0/720692816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c101ea0 msgr2=0x7f383c105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 --2- 172.21.15.73:0/720692816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c101ea0 0x7f383c105ee0 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 crypto rx=0x7f382c009900 tx=0x7f382c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:12:36.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 -- 172.21.15.73:0/720692816 shutdown_connections 2024-03-09T00:12:36.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 --2- 172.21.15.73:0/720692816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 0x7f383c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 --2- 172.21.15.73:0/720692816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c101ea0 0x7f383c105ee0 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 --2- 172.21.15.73:0/720692816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c1018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 -- 172.21.15.73:0/720692816 >> 172.21.15.73:0/720692816 conn(0x7f383c0fcd90 msgr2=0x7f383c0ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 -- 172.21.15.73:0/720692816 shutdown_connections 2024-03-09T00:12:36.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.026+0000 7f3844481640 1 -- 172.21.15.73:0/720692816 wait complete. 2024-03-09T00:12:36.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 Processor -- start 2024-03-09T00:12:36.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 -- start start 2024-03-09T00:12:36.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 0x7f383c19a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c19e650 0x7f383c19eb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c111650 con 0x7f383c19e650 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c1114d0 con 0x7f383c1014f0 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.027+0000 7f3844481640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f383c1117d0 con 0x7f383c1065c0 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f3842a1e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c19e650 0x7f383c19eb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:42320/0 (socket says 172.21.15.73:42320) 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 -- 172.21.15.73:0/4088151454 learned_addr learned my addr 172.21.15.73:0/4088151454 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f3841a1c640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 0x7f383c19a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 -- 172.21.15.73:0/4088151454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 msgr2=0x7f383c19a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 0x7f383c19a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 -- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c19e650 msgr2=0x7f383c19eb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c19e650 0x7f383c19eb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 -- 172.21.15.73:0/4088151454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f383c19f140 con 0x7f383c1014f0 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f384221d640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9f0 tx=0x7f383800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f38337fe640 1 -- 172.21.15.73:0/4088151454 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f383800cd80 con 0x7f383c1014f0 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f383c19f430 con 0x7f383c1014f0 2024-03-09T00:12:36.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f383c1a6c40 con 0x7f383c1014f0 2024-03-09T00:12:36.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f38337fe640 1 -- 172.21.15.73:0/4088151454 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838010910 con 0x7f383c1014f0 2024-03-09T00:12:36.031 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.028+0000 7f38337fe640 1 -- 172.21.15.73:0/4088151454 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838026060 con 0x7f383c1014f0 2024-03-09T00:12:36.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f3847063740 1 Processor -- start 2024-03-09T00:12:36.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f3847063740 1 -- start start 2024-03-09T00:12:36.037 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f3847063740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f3847063740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 0x5603eb1e6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f3847063740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1e9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.031+0000 7f38337fe640 1 -- 172.21.15.73:0/4088151454 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3838010ac0 con 0x7f383c1014f0 2024-03-09T00:12:36.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3847063740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb140e00 con 0x5603eb1de080 2024-03-09T00:12:36.038 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3847063740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb0e9b10 con 0x5603eb1dd760 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3847063740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb13cea0 con 0x5603eb1e6c60 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 0x5603eb1e6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3842a1e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3842a1e640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:42328/0 (socket says 172.21.15.73:42328) 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3842a1e640 1 -- 172.21.15.73:0/1490219985 learned_addr learned my addr 172.21.15.73:0/1490219985 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:36.039 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3841a1c640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1e9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 -- 172.21.15.73:0/1490219985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 msgr2=0x5603eb1e9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1e9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 -- 172.21.15.73:0/1490219985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 msgr2=0x5603eb1ddb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 -- 172.21.15.73:0/1490219985 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603eaf1f7d0 con 0x5603eb1de080 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3841a1c640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:12:36.040 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3842a1e640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f38337fe640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 0x7f38280a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f384221d640 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 0x5603eb1e6720 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f38380cc830 tx=0x7f38380d5ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837176678e6582ca server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f3842a1e640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 0x7f38280a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.032+0000 7f38337fe640 1 -- 172.21.15.73:0/4088151454 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f38380cf470 con 0x7f383c1014f0 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3830bfe640 1 -- 172.21.15.73:0/1490219985 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838097a90 con 0x5603eb1de080 2024-03-09T00:12:36.041 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3830bfe640 1 -- 172.21.15.73:0/1490219985 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838019990 con 0x5603eb1de080 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3830bfe640 1 -- 172.21.15.73:0/1490219985 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3838060fa0 con 0x5603eb1de080 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3847063740 1 -- 172.21.15.73:0/1490219985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 msgr2=0x5603eb1e6720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3847063740 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 0x5603eb1e6720 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 crypto rx=0x7f38380cc830 tx=0x7f38380d5ae0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.033+0000 7f3842a1e640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 0x7f38280a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3834009fd0 tx=0x7f3834009500 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- 172.21.15.73:0/1490219985 shutdown_connections 2024-03-09T00:12:36.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1e9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1de080 0x5603eb1e6720 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- 172.21.15.73:0/1490219985 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1dd760 0x5603eb1ddb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- 172.21.15.73:0/1490219985 >> 172.21.15.73:0/1490219985 conn(0x5603eb1d7e90 msgr2=0x5603eb1d8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- 172.21.15.73:0/1490219985 shutdown_connections 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- 172.21.15.73:0/1490219985 wait complete. 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- start start 2024-03-09T00:12:36.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 Processor -- start 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 0x5603eb1e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e3730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb1e9a50 con 0x5603eb1e6c60 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb0fd0a0 con 0x5603eb1de080 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.034+0000 7f3847063740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5603eb1e9bd0 con 0x5603eb1dd760 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3847063740 10 client.0 _ll_register_callbacks cb 0x5603eb0fca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:36.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3842a1e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f384221d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 0x5603eb1e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e3730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3842a1e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33898/0 (socket says 172.21.15.73:33898) 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f384221d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 0x5603eb1e63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:58784/0 (socket says 172.21.15.73:58784) 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3842a1e640 1 -- 172.21.15.73:0/802666369 learned_addr learned my addr 172.21.15.73:0/802666369 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 -- 172.21.15.73:0/802666369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 msgr2=0x5603eb1e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 0x5603eb1e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 -- 172.21.15.73:0/802666369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 msgr2=0x5603eb1e3c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 -- 172.21.15.73:0/802666369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603eae553f0 con 0x5603eb1de080 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3842a1e640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3841a1c640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e3730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f382c02c170 tx=0x7f382c009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fb4570432aa400a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.035+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c0046c0 con 0x5603eb1de080 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3838019990 con 0x5603eb1de080 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f382c004860 con 0x5603eb1de080 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382c004a60 con 0x5603eb1de080 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 10 client.25925 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5603eae150f0 con 0x5603eb1de080 2024-03-09T00:12:36.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838019990 con 0x5603eb1de080 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f382c0037f0 con 0x5603eb1de080 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 10 client.25925 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 10 client.25925 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 20 client.25925 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.036+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb1f6710 0x5603eb1f8b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.037+0000 7f3847063740 4 client.25925 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.037+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5603eb14e340 con 0x5603eb1f6710 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.037+0000 7f384221d640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb1f6710 0x5603eb1f8b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.037+0000 7f3825ffb640 10 client.25925 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.037+0000 7f384221d640 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb1f6710 0x5603eb1f8b00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d08bfb33ea059f70 server_cookie=a43b00b02a6af886 in_seq=0 out_seq=0 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3825ffb640 1 -- 172.21.15.73:0/802666369 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5603eb14e340 con 0x5603eb1f6710 2024-03-09T00:12:36.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3825ffb640 10 client.25925 handle_command_reply: tid=0 2024-03-09T00:12:36.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 client.25925 shutdown 2024-03-09T00:12:36.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 msgr2=0x5603eb1e3730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e3730 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f382c02c170 tx=0x7f382c009560 comp rx=0 tx=0).stop 2024-03-09T00:12:36.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 shutdown_connections 2024-03-09T00:12:36.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb1f6710 0x5603eb1f8b00 crc :-1 s=READY pgs=143 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1e6c60 0x5603eb1e3c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e3730 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 --2- 172.21.15.73:0/802666369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1dd760 0x5603eb1e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3825ffb640 0 client.25925 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 >> 172.21.15.73:0/802666369 conn(0x5603eb1d7e90 msgr2=0x5603eb1d9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 shutdown_connections 2024-03-09T00:12:36.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 1 -- 172.21.15.73:0/802666369 wait complete. 2024-03-09T00:12:36.052 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.038+0000 7f3847063740 20 client.25925 trim_cache size 0 max 16384 2024-03-09T00:12:36.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 Processor -- start 2024-03-09T00:12:36.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 -- start start 2024-03-09T00:12:36.062 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1e5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ede80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382000de70 con 0x5603eb1dd760 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3820010430 con 0x5603eb1de080 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.055+0000 7f3847063740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382000e470 con 0x5603eb1e6c60 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1e5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.063 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3841a1c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3841a1c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:42338/0 (socket says 172.21.15.73:42338) 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3841a1c640 1 -- 172.21.15.73:0/2258516904 learned_addr learned my addr 172.21.15.73:0/2258516904 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3842a1e640 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ede80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 -- 172.21.15.73:0/2258516904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 msgr2=0x5603eb1ede80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ede80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 -- 172.21.15.73:0/2258516904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 msgr2=0x5603eb1e63a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.064 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 -- 172.21.15.73:0/2258516904 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5603eb1f8e70 con 0x5603eb1dd760 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3841a1c640 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f384221d640 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1e5e60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f383800c6a0 tx=0x7f3838086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84487b0c53ed08b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3827fff640 1 -- 172.21.15.73:0/2258516904 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380636e0 con 0x5603eb1dd760 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3827fff640 1 -- 172.21.15.73:0/2258516904 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838010db0 con 0x5603eb1dd760 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.056+0000 7f3827fff640 1 -- 172.21.15.73:0/2258516904 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38380638b0 con 0x5603eb1dd760 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- 172.21.15.73:0/2258516904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 msgr2=0x5603eb1e5e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.065 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1e5e60 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 crypto rx=0x7f383800c6a0 tx=0x7f3838086040 comp rx=0 tx=0).stop 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- 172.21.15.73:0/2258516904 shutdown_connections 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ede80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1e63a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 --2- 172.21.15.73:0/2258516904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1e5e60 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- 172.21.15.73:0/2258516904 >> 172.21.15.73:0/2258516904 conn(0x5603eb1d7e90 msgr2=0x5603eb1fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- 172.21.15.73:0/2258516904 shutdown_connections 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- 172.21.15.73:0/2258516904 wait complete. 2024-03-09T00:12:36.066 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.057+0000 7f3847063740 1 -- start start 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 Processor -- start 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1edfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ea9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382000eab0 con 0x5603eb1dd760 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382000fdd0 con 0x5603eb1de080 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f382000f110 con 0x5603eb1e6c60 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3847063740 10 client.0 _ll_register_callbacks cb 0x7f382c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:12:36.067 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f384221d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1edfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3841a1c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3841a1c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:42350/0 (socket says 172.21.15.73:42350) 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3841a1c640 1 -- 172.21.15.73:0/3523186754 learned_addr learned my addr 172.21.15.73:0/3523186754 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3842a1e640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ea9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.058+0000 7f3841a1c640 1 -- 172.21.15.73:0/3523186754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 msgr2=0x5603eb1ea9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3841a1c640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ea9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3841a1c640 1 -- 172.21.15.73:0/3523186754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 msgr2=0x5603eb1edfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:12:36.068 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3841a1c640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1edfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3841a1c640 1 -- 172.21.15.73:0/3523186754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5603eb1eaf60 con 0x5603eb1de080 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3841a1c640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f382c009850 tx=0x7f382c02c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5491d9d0149fc378 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3825ffb640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f382c050030 con 0x5603eb1de080 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3838010db0 con 0x5603eb1de080 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f382c0035f0 con 0x5603eb1de080 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f382c003ea0 con 0x5603eb1de080 2024-03-09T00:12:36.069 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3847063740 10 client.25931 fetch_fsmap learned FSMap version 20 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5603eb24da10 con 0x5603eb1de080 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.059+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3838010db0 con 0x5603eb1de080 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f382c047cd0 con 0x5603eb1de080 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 10 client.25931 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 10 client.25931 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 20 client.25931 populate_metadata read hostname 'smithi073' 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb24e0c0 0x5603eb2504e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:12:36.070 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 4 client.25931 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5603eb148690 con 0x5603eb24e0c0 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f384221d640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb24e0c0 0x5603eb2504e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f384221d640 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb24e0c0 0x5603eb2504e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c2bf47812ce5068a server_cookie=97407337b3995ce1 in_seq=0 out_seq=0 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.060+0000 7f3825ffb640 10 client.25931 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3825ffb640 1 -- 172.21.15.73:0/3523186754 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5603eb148690 con 0x5603eb24e0c0 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3825ffb640 10 client.25931 handle_command_reply: tid=0 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 client.25931 shutdown 2024-03-09T00:12:36.071 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 msgr2=0x5603eb1ea460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f382c009850 tx=0x7f382c02c3a0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 shutdown_connections 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5603eb24e0c0 0x5603eb2504e0 crc :-1 s=READY pgs=144 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5603eb1e6c60 0x5603eb1ea9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5603eb1de080 0x5603eb1ea460 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3825ffb640 0 client.25931 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 --2- 172.21.15.73:0/3523186754 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5603eb1dd760 0x5603eb1edfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 >> 172.21.15.73:0/3523186754 conn(0x5603eb1d7e90 msgr2=0x5603eb1de5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.072 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 shutdown_connections 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 1 -- 172.21.15.73:0/3523186754 wait complete. 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.061+0000 7f3847063740 20 client.25931 trim_cache size 0 max 16384 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2387 inodes in the stack)", 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stdout: "31d89c64-3ce3-4b02-ad1c-eb94219b9378": { 2024-03-09T00:12:36.073 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:12:36.074 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "31d89c64-3ce3-4b02-ad1c-eb94219b9378", 2024-03-09T00:12:36.074 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:12:36.074 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:12:36.074 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:12:36.074 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:12:36.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 msgr2=0x7f38280a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 0x7f38280a4e30 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f3834009fd0 tx=0x7f3834009500 comp rx=0 tx=0).stop 2024-03-09T00:12:36.077 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 msgr2=0x7f383c199d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:12:36.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7f383800e9f0 tx=0x7f383800eeb0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 shutdown_connections 2024-03-09T00:12:36.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38280a2970 0x7f38280a4e30 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f383c19e650 0x7f383c19eb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f383c1065c0 0x7f383c19a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 --2- 172.21.15.73:0/4088151454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f383c1014f0 0x7f383c199d90 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:12:36.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 >> 172.21.15.73:0/4088151454 conn(0x7f383c0fcd90 msgr2=0x7f383c0ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:12:36.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 shutdown_connections 2024-03-09T00:12:36.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:12:36.062+0000 7f3844481640 1 -- 172.21.15.73:0/4088151454 wait complete. 2024-03-09T00:12:36.080 INFO:tasks.cephfs.filesystem:scrub status for tag:31d89c64-3ce3-4b02-ad1c-eb94219b9378 - {'path': '/', 'tag': '31d89c64-3ce3-4b02-ad1c-eb94219b9378', 'options': 'recursive,force'} 2024-03-09T00:12:36.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:36 smithi073 ceph-mon[32309]: pgmap v175: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:36.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:36 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:12:36.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:36 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:36.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:36 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:12:36.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:36 smithi102 ceph-mon[36106]: pgmap v175: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:36.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:36 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:12:36.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:36 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:36.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:36 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:12:36.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:36 smithi179 ceph-mon[36637]: pgmap v175: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:36.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:36 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:12:36.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:36 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:36.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:36 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:12:37.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:37 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:37.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:37 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:37.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:37 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:12:38.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:38 smithi073 ceph-mon[32309]: pgmap v176: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 154 op/s 2024-03-09T00:12:38.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:38 smithi102 ceph-mon[36106]: pgmap v176: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 154 op/s 2024-03-09T00:12:38.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:38 smithi179 ceph-mon[36637]: pgmap v176: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 154 KiB/s rd, 154 op/s 2024-03-09T00:12:39.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:39 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:12:39.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:39 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:12:39.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:39 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:12:40.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:40 smithi073 ceph-mon[32309]: pgmap v177: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:40.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:40 smithi102 ceph-mon[36106]: pgmap v177: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:40.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:40 smithi179 ceph-mon[36637]: pgmap v177: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:41.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:41 smithi102 ceph-mon[36106]: pgmap v178: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:41.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:41 smithi179 ceph-mon[36637]: pgmap v178: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:41.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:41 smithi073 ceph-mon[32309]: pgmap v178: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:44.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:44 smithi073 ceph-mon[32309]: pgmap v179: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:44.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:44 smithi102 ceph-mon[36106]: pgmap v179: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:44.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:44 smithi179 ceph-mon[36637]: pgmap v179: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:46.386 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:46 smithi179 ceph-mon[36637]: pgmap v180: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:46.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:46 smithi073 ceph-mon[32309]: pgmap v180: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:46.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:46 smithi102 ceph-mon[36106]: pgmap v180: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:12:48.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:48 smithi073 ceph-mon[32309]: pgmap v181: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:48.510 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:48 smithi102 ceph-mon[36106]: pgmap v181: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:48.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:48 smithi179 ceph-mon[36637]: pgmap v181: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:12:50.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:50 smithi073 ceph-mon[32309]: pgmap v182: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:12:50.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:50 smithi102 ceph-mon[36106]: pgmap v182: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:12:50.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:50 smithi179 ceph-mon[36637]: pgmap v182: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:12:52.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:52 smithi073 ceph-mon[32309]: pgmap v183: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:52.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:52 smithi102 ceph-mon[36106]: pgmap v183: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:52.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:52 smithi179 ceph-mon[36637]: pgmap v183: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:54.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:54 smithi073 ceph-mon[32309]: pgmap v184: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:54.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:54 smithi102 ceph-mon[36106]: pgmap v184: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:54.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:54 smithi179 ceph-mon[36637]: pgmap v184: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:56 smithi073 ceph-mon[32309]: pgmap v185: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:56 smithi102 ceph-mon[36106]: pgmap v185: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:56.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:56 smithi179 ceph-mon[36637]: pgmap v185: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:12:58.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:12:58 smithi073 ceph-mon[32309]: pgmap v186: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:12:58 smithi102 ceph-mon[36106]: pgmap v186: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:12:58.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:12:58 smithi179 ceph-mon[36637]: pgmap v186: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:00.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:00 smithi073 ceph-mon[32309]: pgmap v187: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:00.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:00 smithi102 ceph-mon[36106]: pgmap v187: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:00.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:00 smithi179 ceph-mon[36637]: pgmap v187: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:01.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:01 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:13:01.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:01 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:13:01.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:01 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:13:02.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:02 smithi073 ceph-mon[32309]: pgmap v188: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:02.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:13:02.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:13:02.407 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:13:02.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:02 smithi102 ceph-mon[36106]: pgmap v188: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:02.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:13:02.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:13:02.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:13:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:02 smithi179 ceph-mon[36637]: pgmap v188: 129 pgs: 129 active+clean; 35 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:02.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:13:02.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:13:02.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:13:04.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:04 smithi073 ceph-mon[32309]: pgmap v189: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:04 smithi102 ceph-mon[36106]: pgmap v189: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:04.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:04 smithi179 ceph-mon[36637]: pgmap v189: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:06.081 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:13:06.081 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:13:06.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/2158385130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68105550 msgr2=0x7fdd6810d120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/2158385130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68105550 0x7fdd6810d120 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009ca0 tx=0x7fdd6002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/2158385130 shutdown_connections 2024-03-09T00:13:06.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/2158385130 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68105550 0x7fdd6810d120 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/2158385130 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd68100e30 0x7fdd68104e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/2158385130 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68100480 0x7fdd68100860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/2158385130 >> 172.21.15.73:0/2158385130 conn(0x7fdd68077350 msgr2=0x7fdd68077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/2158385130 shutdown_connections 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/2158385130 wait complete. 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 Processor -- start 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.201+0000 7fdd6fbe1640 1 -- start start 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 0x7fdd6819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681979a0 0x7fdd68197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68110610 con 0x7fdd68100480 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68110490 con 0x7fdd68105550 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd68110790 con 0x7fdd681979a0 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33484/0 (socket says 172.21.15.73:33484) 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 -- 172.21.15.73:0/3833263269 learned_addr learned my addr 172.21.15.73:0/3833263269 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 0x7fdd6819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 -- 172.21.15.73:0/3833263269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681979a0 msgr2=0x7fdd68197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681979a0 0x7fdd68197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 -- 172.21.15.73:0/3833263269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 msgr2=0x7fdd6819dd30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 0x7fdd6819dd30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 -- 172.21.15.73:0/3833263269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd681984c0 con 0x7fdd68100480 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 0x7fdd6819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6d97d640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800e9f0 tx=0x7fdd5800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd5effd640 1 -- 172.21.15.73:0/3833263269 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd5800cd80 con 0x7fdd68100480 2024-03-09T00:13:06.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd5effd640 1 -- 172.21.15.73:0/3833263269 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd58010910 con 0x7fdd68100480 2024-03-09T00:13:06.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd681987b0 con 0x7fdd68100480 2024-03-09T00:13:06.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd5effd640 1 -- 172.21.15.73:0/3833263269 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd58018990 con 0x7fdd68100480 2024-03-09T00:13:06.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.202+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd681a4a00 con 0x7fdd68100480 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.204+0000 7fdd5effd640 1 -- 172.21.15.73:0/3833263269 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fdd58010ac0 con 0x7fdd68100480 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 Processor -- start 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 -- start start 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04747760 0x560f04747b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04750720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd5effd640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 0x7fdd540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.205+0000 7fdd727bd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f046aae00 con 0x560f04750c60 2024-03-09T00:13:06.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd727bd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f04653b10 con 0x560f04747760 2024-03-09T00:13:06.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd727bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f046a6ea0 con 0x560f04748080 2024-03-09T00:13:06.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6d97d640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 0x7fdd540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33496/0 (socket says 172.21.15.73:33496) 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 -- 172.21.15.73:0/4238502757 learned_addr learned my addr 172.21.15.73:0/4238502757 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04747760 0x560f04747b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04750720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd5effd640 1 -- 172.21.15.73:0/3833263269 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fdd580cf6b0 con 0x7fdd68100480 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 -- 172.21.15.73:0/4238502757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 msgr2=0x560f04750720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04750720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 -- 172.21.15.73:0/4238502757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04747760 msgr2=0x560f04747b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04747760 0x560f04747b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 -- 172.21.15.73:0/4238502757 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f044897d0 con 0x560f04750c60 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04750720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6d97d640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 0x7fdd540a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd58098350 tx=0x7fdd5806d670 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdd60004af0 tx=0x7fdd60004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e62509677aaac99 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd53fff640 1 -- 172.21.15.73:0/4238502757 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60039040 con 0x560f04750c60 2024-03-09T00:13:06.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd53fff640 1 -- 172.21.15.73:0/4238502757 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd60032c90 con 0x560f04750c60 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.206+0000 7fdd53fff640 1 -- 172.21.15.73:0/4238502757 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60004030 con 0x560f04750c60 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 -- 172.21.15.73:0/4238502757 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 msgr2=0x560f04753050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 secure :-1 s=READY pgs=828 cs=0 l=1 rev1=1 crypto rx=0x7fdd60004af0 tx=0x7fdd60004a60 comp rx=0 tx=0).stop 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 -- 172.21.15.73:0/4238502757 shutdown_connections 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f04753050 unknown :-1 s=CLOSED pgs=828 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04750720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4238502757 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04747760 0x560f04747b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 -- 172.21.15.73:0/4238502757 >> 172.21.15.73:0/4238502757 conn(0x560f04741e90 msgr2=0x560f04742270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 -- 172.21.15.73:0/4238502757 shutdown_connections 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.207+0000 7fdd727bd740 1 -- 172.21.15.73:0/4238502757 wait complete. 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 -- start start 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 Processor -- start 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f04750420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f0474dcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f047539f0 con 0x560f04748080 2024-03-09T00:13:06.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f046670a0 con 0x560f04750c60 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x560f04753b70 con 0x560f04747760 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd727bd740 10 client.0 _ll_register_callbacks cb 0x560f04666a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd6d17c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd6d17c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33512/0 (socket says 172.21.15.73:33512) 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd6d17c640 1 -- 172.21.15.73:0/4280883662 learned_addr learned my addr 172.21.15.73:0/4280883662 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.208+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f04750420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f0474dcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 -- 172.21.15.73:0/4280883662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 msgr2=0x560f04750420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f04750420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 -- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 msgr2=0x560f0474dcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f0474dcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 -- 172.21.15.73:0/4280883662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f043bf3f0 con 0x560f04748080 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f0474dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdd640124a0 tx=0x7fdd64012960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=612735fd25f7be08 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd6401f060 con 0x560f04748080 2024-03-09T00:13:06.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fdd64004460 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd60032c90 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd64026050 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 10 client.16659 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560f0437f0f0 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd60032c90 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdd64019030 con 0x560f04748080 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 10 client.16659 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:06.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 10 client.16659 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 20 client.16659 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.209+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f04760760 0x560f04762ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.210+0000 7fdd727bd740 4 client.16659 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.210+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560f046b8340 con 0x560f04760760 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.210+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f04760760 0x560f04762ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.210+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f04760760 0x560f04762ba0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8929d7ce8f89656e server_cookie=a4df883b8b568a02 in_seq=0 out_seq=0 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.210+0000 7fdd517fa640 10 client.16659 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd517fa640 1 -- 172.21.15.73:0/4280883662 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x560f046b8340 con 0x560f04760760 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd517fa640 10 client.16659 handle_command_reply: tid=0 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 client.16659 shutdown 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 msgr2=0x560f0474d7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 crypto rx=0x7fdd640124a0 tx=0x7fdd64012960 comp rx=0 tx=0).stop 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 shutdown_connections 2024-03-09T00:13:06.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f04760760 0x560f04762ba0 crc :-1 s=READY pgs=145 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f0474dcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04748080 0x560f0474d7b0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 --2- 172.21.15.73:0/4280883662 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f04750420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd517fa640 0 client.16659 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 >> 172.21.15.73:0/4280883662 conn(0x560f04741e90 msgr2=0x560f04743be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 shutdown_connections 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 1 -- 172.21.15.73:0/4280883662 wait complete. 2024-03-09T00:13:06.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.211+0000 7fdd727bd740 20 client.16659 trim_cache size 0 max 16384 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 Processor -- start 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 -- start start 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f0474fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04748080 0x560f04750420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681ac710 con 0x560f04750c60 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681aecd0 con 0x560f04748080 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.226+0000 7fdd727bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681acd10 con 0x560f04747760 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33526/0 (socket says 172.21.15.73:33526) 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3516933668 learned_addr learned my addr 172.21.15.73:0/3516933668 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3516933668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 msgr2=0x560f0474fee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f0474fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3516933668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04748080 msgr2=0x560f04750420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04748080 0x560f04750420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3516933668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560f04747eb0 con 0x560f04750c60 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009dc0 tx=0x7fdd60004a60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=881210ea93e33a67 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd537fe640 1 -- 172.21.15.73:0/3516933668 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60057070 con 0x560f04750c60 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd537fe640 1 -- 172.21.15.73:0/3516933668 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd60032c90 con 0x560f04750c60 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.227+0000 7fdd537fe640 1 -- 172.21.15.73:0/3516933668 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60039040 con 0x560f04750c60 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- 172.21.15.73:0/3516933668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 msgr2=0x560f0474c080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 crypto rx=0x7fdd60009dc0 tx=0x7fdd60004a60 comp rx=0 tx=0).stop 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- 172.21.15.73:0/3516933668 shutdown_connections 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04750c60 0x560f0474c080 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04748080 0x560f04750420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3516933668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04747760 0x560f0474fee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- 172.21.15.73:0/3516933668 >> 172.21.15.73:0/3516933668 conn(0x560f04741e90 msgr2=0x560f0474d770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- 172.21.15.73:0/3516933668 shutdown_connections 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- 172.21.15.73:0/3516933668 wait complete. 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- start start 2024-03-09T00:13:06.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 Processor -- start 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04756510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f04756a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681ad350 con 0x560f04747760 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681ae670 con 0x560f04750c60 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd681ad9b0 con 0x560f04748080 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.228+0000 7fdd727bd740 10 client.0 _ll_register_callbacks cb 0x7fdd64009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:06.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33534/0 (socket says 172.21.15.73:33534) 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3799382859 learned_addr learned my addr 172.21.15.73:0/3799382859 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f04756a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3799382859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 msgr2=0x560f04756510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04756510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 msgr2=0x560f04756a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f04756a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 -- 172.21.15.73:0/3799382859 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560f04757260 con 0x560f04747760 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd6e17e640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000f80 tx=0x7fdd60009540 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3be7230e97c19015 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd517fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd60057070 con 0x560f04747760 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fdd60034440 con 0x560f04747760 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdd60032c90 con 0x560f04747760 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdd60039040 con 0x560f04747760 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd727bd740 10 client.16665 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:06.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x560f047b7d20 con 0x560f04747760 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.229+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd60032c90 con 0x560f04747760 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fdd6003f050 con 0x560f04747760 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 10 client.16665 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 10 client.16665 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 20 client.16665 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f047b7f50 0x560f047ba340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 4 client.16665 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560f0475d030 con 0x560f047b7f50 2024-03-09T00:13:06.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f047b7f50 0x560f047ba340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd6d17c640 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f047b7f50 0x560f047ba340 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2591c152f21004d4 server_cookie=62fde439892f12e8 in_seq=0 out_seq=0 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd517fa640 10 client.16665 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd517fa640 1 -- 172.21.15.73:0/3799382859 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x560f0475d030 con 0x560f047b7f50 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd517fa640 10 client.16665 handle_command_reply: tid=0 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.230+0000 7fdd727bd740 1 client.16665 shutdown 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 msgr2=0x560f04758170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 crypto rx=0x7fdd60000f80 tx=0x7fdd60009540 comp rx=0 tx=0).stop 2024-03-09T00:13:06.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 shutdown_connections 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x560f047b7f50 0x560f047ba340 crc :-1 s=READY pgs=146 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x560f04750c60 0x560f04756a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x560f04748080 0x560f04756510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd517fa640 0 client.16665 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 --2- 172.21.15.73:0/3799382859 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x560f04747760 0x560f04758170 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 >> 172.21.15.73:0/3799382859 conn(0x560f04741e90 msgr2=0x560f0474bfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 shutdown_connections 2024-03-09T00:13:06.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 1 -- 172.21.15.73:0/3799382859 wait complete. 2024-03-09T00:13:06.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd727bd740 20 client.16665 trim_cache size 0 max 16384 2024-03-09T00:13:06.239 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:06.239 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:13:06.239 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:13:06.239 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 msgr2=0x7fdd540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 0x7fdd540a4e30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7fdd58098350 tx=0x7fdd5806d670 comp rx=0 tx=0).stop 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 msgr2=0x7fdd6819d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 crypto rx=0x7fdd5800e9f0 tx=0x7fdd5800eeb0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 shutdown_connections 2024-03-09T00:13:06.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdd681979a0 0x7fdd68197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdd68105550 0x7fdd6819dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdd540a2970 0x7fdd540a4e30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 --2- 172.21.15.73:0/3833263269 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdd68100480 0x7fdd6819d7f0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 >> 172.21.15.73:0/3833263269 conn(0x7fdd68077350 msgr2=0x7fdd681041f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 shutdown_connections 2024-03-09T00:13:06.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.231+0000 7fdd6fbe1640 1 -- 172.21.15.73:0/3833263269 wait complete. 2024-03-09T00:13:06.242 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:13:06.243 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:13:06.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 -- 172.21.15.73:0/3432801189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1810b8c0 msgr2=0x7f9d1810dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 --2- 172.21.15.73:0/3432801189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1810b8c0 0x7f9d1810dd30 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 crypto rx=0x7f9d10009ca0 tx=0x7f9d1002c180 comp rx=0 tx=0).stop 2024-03-09T00:13:06.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 -- 172.21.15.73:0/3432801189 shutdown_connections 2024-03-09T00:13:06.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 --2- 172.21.15.73:0/3432801189 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d1810b8c0 0x7f9d1810dd30 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.347 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 --2- 172.21.15.73:0/3432801189 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d18102ea0 0x7f9d1810b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 --2- 172.21.15.73:0/3432801189 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102580 0x7f9d18102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 -- 172.21.15.73:0/3432801189 >> 172.21.15.73:0/3432801189 conn(0x7f9d18077350 msgr2=0x7f9d18077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 -- 172.21.15.73:0/3432801189 shutdown_connections 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.347+0000 7f9d1e844640 1 -- 172.21.15.73:0/3432801189 wait complete. 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 Processor -- start 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 -- start start 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.348 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102ea0 0x7f9d18100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 0x7f9d18100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d18111670 con 0x7f9d18102580 2024-03-09T00:13:06.349 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181114f0 con 0x7f9d1810b8c0 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d1e844640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d181117f0 con 0x7f9d18102ea0 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d17fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33554/0 (socket says 172.21.15.73:33554) 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d17fff640 1 -- 172.21.15.73:0/2387482816 learned_addr learned my addr 172.21.15.73:0/2387482816 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.348+0000 7f9d177fe640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102ea0 0x7f9d18100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d1cde1640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 0x7f9d18100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 -- 172.21.15.73:0/2387482816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102ea0 msgr2=0x7f9d18100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.350 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102ea0 0x7f9d18100710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 -- 172.21.15.73:0/2387482816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 msgr2=0x7f9d18100c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 0x7f9d18100c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 -- 172.21.15.73:0/2387482816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d181a5fb0 con 0x7f9d18102580 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d1cde1640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 0x7f9d18100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d17fff640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800e9f0 tx=0x7f9d0800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d0800cd80 con 0x7f9d18102580 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d08005500 con 0x7f9d18102580 2024-03-09T00:13:06.351 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d181a62a0 con 0x7f9d18102580 2024-03-09T00:13:06.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d08010690 con 0x7f9d18102580 2024-03-09T00:13:06.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.349+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d181a67a0 con 0x7f9d18102580 2024-03-09T00:13:06.352 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.350+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d18068530 con 0x7f9d18102580 2024-03-09T00:13:06.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.351+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9d080046c0 con 0x7f9d18102580 2024-03-09T00:13:06.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.352+0000 7f9d157fa640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 0x7f9d000a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.352+0000 7f9d177fe640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 0x7f9d000a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.353 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.352+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f9d080cf710 con 0x7f9d18102580 2024-03-09T00:13:06.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.352+0000 7f9d177fe640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 0x7f9d000a4e60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c005fd0 tx=0x7f9d0c005850 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.354 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.356+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9d0809c0e0 con 0x7f9d18102580 2024-03-09T00:13:06.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:06 smithi073 ceph-mon[32309]: pgmap v190: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:06.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:06 smithi102 ceph-mon[36106]: pgmap v190: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:06.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.567+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d181016b0 con 0x7f9d18102580 2024-03-09T00:13:06.564 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.567+0000 7f9d157fa640 1 -- 172.21.15.73:0/2387482816 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f9d0809b9e0 con 0x7f9d18102580 2024-03-09T00:13:06.565 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:13:06.565 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:13:06.565 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 msgr2=0x7f9d000a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 0x7f9d000a4e60 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9d0c005fd0 tx=0x7f9d0c005850 comp rx=0 tx=0).stop 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 msgr2=0x7f9d181001d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 crypto rx=0x7f9d0800e9f0 tx=0x7f9d0800edf0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 shutdown_connections 2024-03-09T00:13:06.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9d1810b8c0 0x7f9d18100c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9d18102ea0 0x7f9d18100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9d000a29a0 0x7f9d000a4e60 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 --2- 172.21.15.73:0/2387482816 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9d18102580 0x7f9d181001d0 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 >> 172.21.15.73:0/2387482816 conn(0x7f9d18077350 msgr2=0x7f9d180fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 shutdown_connections 2024-03-09T00:13:06.568 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.570+0000 7f9d1e844640 1 -- 172.21.15.73:0/2387482816 wait complete. 2024-03-09T00:13:06.578 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:13:06.578 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:13:06.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:06 smithi179 ceph-mon[36637]: pgmap v190: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:06.695 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 -- 172.21.15.73:0/1732658991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce01062d0 msgr2=0x7f6ce01066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/1732658991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce01062d0 0x7f6ce01066b0 secure :-1 s=READY pgs=834 cs=0 l=1 rev1=1 crypto rx=0x7f6cdc009900 tx=0x7f6cdc02bc80 comp rx=0 tx=0).stop 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 -- 172.21.15.73:0/1732658991 shutdown_connections 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/1732658991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100e00 0x7f6ce010d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/1732658991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce0100440 0x7f6ce01008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/1732658991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce01062d0 0x7f6ce01066b0 unknown :-1 s=CLOSED pgs=834 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 -- 172.21.15.73:0/1732658991 >> 172.21.15.73:0/1732658991 conn(0x7f6ce0076fd0 msgr2=0x7f6ce00773e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 -- 172.21.15.73:0/1732658991 shutdown_connections 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.684+0000 7f6ce6e32640 1 -- 172.21.15.73:0/1732658991 wait complete. 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 Processor -- start 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 -- start start 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100440 0x7f6ce019c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce01062d0 0x7f6ce01968d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01105b0 con 0x7f6ce0100e00 2024-03-09T00:13:06.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0110430 con 0x7f6ce01062d0 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce6e32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce0110730 con 0x7f6ce0100440 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33560/0 (socket says 172.21.15.73:33560) 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3896815977 learned_addr learned my addr 172.21.15.73:0/3896815977 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100440 0x7f6ce019c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6ce53cf640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce01062d0 0x7f6ce01968d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3896815977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100440 msgr2=0x7f6ce019c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100440 0x7f6ce019c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3896815977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce01062d0 msgr2=0x7f6ce01968d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce01062d0 0x7f6ce01968d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3896815977 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0197050 con 0x7f6ce0100e00 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000d950 tx=0x7f6cd000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.685+0000 7f6cd5ffb640 1 -- 172.21.15.73:0/3896815977 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0017070 con 0x7f6ce0100e00 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.686+0000 7f6cd5ffb640 1 -- 172.21.15.73:0/3896815977 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd0009c30 con 0x7f6ce0100e00 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.686+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce0197340 con 0x7f6ce0100e00 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.686+0000 7f6cd5ffb640 1 -- 172.21.15.73:0/3896815977 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd000f6b0 con 0x7f6ce0100e00 2024-03-09T00:13:06.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.686+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce01a3590 con 0x7f6ce0100e00 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.688+0000 7f6cd5ffb640 1 -- 172.21.15.73:0/3896815977 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6cd00109d0 con 0x7f6ce0100e00 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 Processor -- start 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 -- start start 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea964580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd5ffb640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 0x7f6cc80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea96fa90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea92b360 con 0x5607ea964ac0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea8c7ec0 con 0x5607ea9641a0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea870d30 con 0x5607ea96d6a0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 0x7f6cc80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33576/0 (socket says 172.21.15.73:33576) 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 -- 172.21.15.73:0/741065307 learned_addr learned my addr 172.21.15.73:0/741065307 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd5ffb640 1 -- 172.21.15.73:0/3896815977 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6cd00cedd0 con 0x7f6ce0100e00 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce53cf640 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea964580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce53cf640 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea96fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 -- 172.21.15.73:0/741065307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 msgr2=0x5607ea96fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea96fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 -- 172.21.15.73:0/741065307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 msgr2=0x5607ea964580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea964580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 -- 172.21.15.73:0/741065307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ea86fbc0 con 0x5607ea964ac0 2024-03-09T00:13:06.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6cd009b030 tx=0x7f6cd00662b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa9e279d6000df3c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.689+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 0x7f6cc80a4eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0105400 tx=0x7f6cdc005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6cc6ffd640 1 -- 172.21.15.73:0/741065307 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0020070 con 0x5607ea964ac0 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6cc6ffd640 1 -- 172.21.15.73:0/741065307 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd0009de0 con 0x5607ea964ac0 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6cc6ffd640 1 -- 172.21.15.73:0/741065307 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00993a0 con 0x5607ea964ac0 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/741065307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 msgr2=0x5607ea96d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 crypto rx=0x7f6cd009b030 tx=0x7f6cd00662b0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/741065307 shutdown_connections 2024-03-09T00:13:06.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea96fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea96d160 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/741065307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea964580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/741065307 >> 172.21.15.73:0/741065307 conn(0x5607ea95e8d0 msgr2=0x5607ea95ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/741065307 shutdown_connections 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.690+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/741065307 wait complete. 2024-03-09T00:13:06.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 -- start start 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 Processor -- start 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea9641a0 0x5607ea9706c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea96d6a0 0x5607ea971140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea8664a0 con 0x5607ea964ac0 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea8c3f60 con 0x5607ea96d6a0 2024-03-09T00:13:06.705 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607ea970430 con 0x5607ea9641a0 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce9a0e740 10 client.0 _ll_register_callbacks cb 0x5607ea8c1d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33586/0 (socket says 172.21.15.73:33586) 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 -- 172.21.15.73:0/612611809 learned_addr learned my addr 172.21.15.73:0/612611809 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea9641a0 0x5607ea9706c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.706 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6ce53cf640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea96d6a0 0x5607ea971140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 -- 172.21.15.73:0/612611809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea9641a0 msgr2=0x5607ea9706c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea9641a0 0x5607ea9706c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 -- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea96d6a0 msgr2=0x5607ea971140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea96d6a0 0x5607ea971140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.691+0000 7f6cd7fff640 1 -- 172.21.15.73:0/612611809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607ea5dc3f0 con 0x5607ea964ac0 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0010fd0 tx=0x7f6cd0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5941c3dd743e58ae server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:06.707 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0020040 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6cd0009de0 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6cd0097710 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cd009f1f0 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 10 client.16683 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5607ea6a67d0 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd0009de0 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6cd009a5d0 con 0x5607ea964ac0 2024-03-09T00:13:06.708 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 10 client.16683 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 10 client.16683 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 20 client.16683 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea97f8e0 0x5607ea981cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 4 client.16683 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.692+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607ea8d5400 con 0x5607ea97f8e0 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.693+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea97f8e0 0x5607ea981cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.693+0000 7f6cbffff640 10 client.16683 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.693+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea97f8e0 0x5607ea981cd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2738649bb1c952fa server_cookie=703f3eaf4f5da89d in_seq=0 out_seq=0 2024-03-09T00:13:06.709 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6cbffff640 1 -- 172.21.15.73:0/612611809 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5607ea8d5400 con 0x5607ea97f8e0 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6cbffff640 10 client.16683 handle_command_reply: tid=0 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 client.16683 shutdown 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 msgr2=0x5607ea970c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 secure :-1 s=READY pgs=837 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0010fd0 tx=0x7f6cd0000bc0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 shutdown_connections 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea97f8e0 0x5607ea981cd0 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea96d6a0 0x5607ea971140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6cbffff640 0 client.16683 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.710 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea970c00 unknown :-1 s=CLOSED pgs=837 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/612611809 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea9641a0 0x5607ea9706c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 >> 172.21.15.73:0/612611809 conn(0x5607ea95e8d0 msgr2=0x5607ea960620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 shutdown_connections 2024-03-09T00:13:06.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/612611809 wait complete. 2024-03-09T00:13:06.711 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.694+0000 7f6ce9a0e740 20 client.16683 trim_cache size 0 max 16384 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.705+0000 7f6ce9a0e740 1 Processor -- start 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.705+0000 7f6ce9a0e740 1 -- start start 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea97b190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea9756c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.717 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01aa4c0 con 0x5607ea964ac0 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01aca80 con 0x5607ea9641a0 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01aaac0 con 0x5607ea96d6a0 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33598/0 (socket says 172.21.15.73:33598) 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3311206212 learned_addr learned my addr 172.21.15.73:0/3311206212 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea97b190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3311206212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 msgr2=0x5607ea9756c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.718 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea9756c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3311206212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 msgr2=0x5607ea97b190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea97b190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 -- 172.21.15.73:0/3311206212 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607ea958da0 con 0x5607ea964ac0 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00cc790 tx=0x7f6cd00995b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=98f58414d9915583 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cc67fc640 1 -- 172.21.15.73:0/3311206212 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0020030 con 0x5607ea964ac0 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cc67fc640 1 -- 172.21.15.73:0/3311206212 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd00bb030 con 0x5607ea964ac0 2024-03-09T00:13:06.719 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.706+0000 7f6cc67fc640 1 -- 172.21.15.73:0/3311206212 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd00107e0 con 0x5607ea964ac0 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/3311206212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 msgr2=0x5607ea9862c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 crypto rx=0x7f6cd00cc790 tx=0x7f6cd00995b0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/3311206212 shutdown_connections 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea9756c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea9862c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/3311206212 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea97b190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/3311206212 >> 172.21.15.73:0/3311206212 conn(0x5607ea95e8d0 msgr2=0x5607ea95ecb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/3311206212 shutdown_connections 2024-03-09T00:13:06.720 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/3311206212 wait complete. 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 -- start start 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 Processor -- start 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea971ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.707+0000 7f6ce9a0e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea972a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01ab100 con 0x5607ea964ac0 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01ac420 con 0x5607ea9641a0 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce01ab760 con 0x5607ea96d6a0 2024-03-09T00:13:06.721 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 10 client.0 _ll_register_callbacks cb 0x7f6cd000df40 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33602/0 (socket says 172.21.15.73:33602) 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 -- 172.21.15.73:0/2445604996 learned_addr learned my addr 172.21.15.73:0/2445604996 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea971ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 -- 172.21.15.73:0/2445604996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 msgr2=0x5607ea972a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea972a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.722 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 -- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 msgr2=0x5607ea971ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea971ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 -- 172.21.15.73:0/2445604996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607ea9d7e40 con 0x5607ea964ac0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cd7fff640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0099350 tx=0x7f6cd00979a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a5cf5fb70bf8c5d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cbffff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6cd0020040 con 0x5607ea964ac0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7f6cd00b9da0 con 0x5607ea964ac0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6cd00bb030 con 0x5607ea964ac0 2024-03-09T00:13:06.723 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6cd0009de0 con 0x5607ea964ac0 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 10 client.16689 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5607ea9d7e40 con 0x5607ea964ac0 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.708+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6cd00bb030 con 0x5607ea964ac0 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f6cd009a160 con 0x5607ea964ac0 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 10 client.16689 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 10 client.16689 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 20 client.16689 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea9d9260 0x5607ea9db650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:06.724 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 4 client.16689 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5607ea8cf750 con 0x5607ea9d9260 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea9d9260 0x5607ea9db650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce4bce640 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea9d9260 0x5607ea9db650 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9be881af8b94b6b2 server_cookie=49ca3de4d22323d6 in_seq=0 out_seq=0 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6cbffff640 10 client.16689 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6cbffff640 1 -- 172.21.15.73:0/2445604996 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5607ea8cf750 con 0x5607ea9d9260 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6cbffff640 10 client.16689 handle_command_reply: tid=0 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.709+0000 7f6ce9a0e740 1 client.16689 shutdown 2024-03-09T00:13:06.725 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 msgr2=0x5607ea972530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 crypto rx=0x7f6cd0099350 tx=0x7f6cd00979a0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 shutdown_connections 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5607ea9d9260 0x5607ea9db650 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5607ea96d6a0 0x5607ea972a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6cbffff640 0 client.16689 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5607ea964ac0 0x5607ea972530 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 --2- 172.21.15.73:0/2445604996 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5607ea9641a0 0x5607ea971ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.726 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 >> 172.21.15.73:0/2445604996 conn(0x5607ea95e8d0 msgr2=0x5607ea968b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.727 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 shutdown_connections 2024-03-09T00:13:06.727 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 1 -- 172.21.15.73:0/2445604996 wait complete. 2024-03-09T00:13:06.727 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce9a0e740 20 client.16689 trim_cache size 0 max 16384 2024-03-09T00:13:06.727 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:13:06.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 msgr2=0x7f6cc80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 0x7f6cc80a4eb0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f6ce0105400 tx=0x7f6cdc005fd0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 msgr2=0x7f6ce019cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:06.729 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 crypto rx=0x7f6cd000d950 tx=0x7f6cd000de10 comp rx=0 tx=0).stop 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 shutdown_connections 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f6ce01062d0 0x7f6ce01968d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6cc80a29f0 0x7f6cc80a4eb0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6ce0100e00 0x7f6ce019cc10 unknown :-1 s=CLOSED pgs=835 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 --2- 172.21.15.73:0/3896815977 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6ce0100440 0x7f6ce019c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 >> 172.21.15.73:0/3896815977 conn(0x7f6ce0076fd0 msgr2=0x7f6ce00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 shutdown_connections 2024-03-09T00:13:06.730 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:06.710+0000 7f6ce6e32640 1 -- 172.21.15.73:0/3896815977 wait complete. 2024-03-09T00:13:07.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2387482816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:07.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2387482816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:07.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2387482816' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:07.731 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:13:07.732 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:13:07.732 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:13:07.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/4039136747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 msgr2=0x7fa5b410b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/4039136747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b410b330 secure :-1 s=READY pgs=840 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4009900 tx=0x7fa5a402bcd0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/4039136747 shutdown_connections 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/4039136747 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b410b900 0x7fa5b410dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/4039136747 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b410b330 unknown :-1 s=CLOSED pgs=840 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/4039136747 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b4102530 0x7fa5b4102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/4039136747 >> 172.21.15.73:0/4039136747 conn(0x7fa5b4076570 msgr2=0x7fa5b4076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/4039136747 shutdown_connections 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/4039136747 wait complete. 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 Processor -- start 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.853+0000 7fa5bc6c6640 1 -- start start 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 0x7fa5b4100180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 0x7fa5b4100c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4111690 con 0x7fa5b4102e50 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4111510 con 0x7fa5b4102530 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b4111810 con 0x7fa5b410b900 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33616/0 (socket says 172.21.15.73:33616) 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3942692008 learned_addr learned my addr 172.21.15.73:0/3942692008 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 0x7fa5b4100c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5ba462640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 0x7fa5b4100180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3942692008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 msgr2=0x7fa5b4100c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 0x7fa5b4100c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3942692008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 msgr2=0x7fa5b4100180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 0x7fa5b4100180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3942692008 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5b41a5ff0 con 0x7fa5b4102e50 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5ba462640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 0x7fa5b4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 0x7fa5b4100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4005940 tx=0x7fa5a402cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5ab7fe640 1 -- 172.21.15.73:0/3942692008 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4041070 con 0x7fa5b4102e50 2024-03-09T00:13:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5b41a6280 con 0x7fa5b4102e50 2024-03-09T00:13:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5ab7fe640 1 -- 172.21.15.73:0/3942692008 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5a4004450 con 0x7fa5b4102e50 2024-03-09T00:13:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5ab7fe640 1 -- 172.21.15.73:0/3942692008 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a4034440 con 0x7fa5b4102e50 2024-03-09T00:13:07.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.854+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5b41a67c0 con 0x7fa5b4102e50 2024-03-09T00:13:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5ab7fe640 1 -- 172.21.15.73:0/3942692008 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa5a4044070 con 0x7fa5b4102e50 2024-03-09T00:13:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 Processor -- start 2024-03-09T00:13:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 -- start start 2024-03-09T00:13:07.865 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c486d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c489670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c3e1b40 con 0x56378c47dd80 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c38ad80 con 0x56378c47e6a0 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5bf386740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c3ddbe0 con 0x56378c487280 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.857+0000 7fa5ab7fe640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 0x7fa5a00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.866 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33624/0 (socket says 172.21.15.73:33624) 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 -- 172.21.15.73:0/3633878367 learned_addr learned my addr 172.21.15.73:0/3633878367 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5ab7fe640 1 -- 172.21.15.73:0/3942692008 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa5a40f0da0 con 0x7fa5b4102e50 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 0x7fa5a00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c489670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.867 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5ba462640 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c486d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 -- 172.21.15.73:0/3633878367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 msgr2=0x56378c489670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c489670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 -- 172.21.15.73:0/3633878367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 msgr2=0x56378c486d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c486d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 -- 172.21.15.73:0/3633878367 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56378c1be7e0 con 0x56378c47dd80 2024-03-09T00:13:07.868 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac013e30 tx=0x7fa5ac0107a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61223020928436c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 0x7fa5a00a4e60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac010ca0 tx=0x7fa5ac015000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5a8ff9640 1 -- 172.21.15.73:0/3633878367 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac021070 con 0x56378c47dd80 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5a8ff9640 1 -- 172.21.15.73:0/3633878367 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5ac007bb0 con 0x56378c47dd80 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.858+0000 7fa5a8ff9640 1 -- 172.21.15.73:0/3633878367 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac0057a0 con 0x56378c47dd80 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- 172.21.15.73:0/3633878367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 msgr2=0x56378c47e160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac013e30 tx=0x7fa5ac0107a0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.869 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- 172.21.15.73:0/3633878367 shutdown_connections 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c489670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c486d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3633878367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c47e160 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- 172.21.15.73:0/3633878367 >> 172.21.15.73:0/3633878367 conn(0x56378c4784b0 msgr2=0x56378c478890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- 172.21.15.73:0/3633878367 shutdown_connections 2024-03-09T00:13:07.870 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- 172.21.15.73:0/3633878367 wait complete. 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 -- start start 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.859+0000 7fa5bf386740 1 Processor -- start 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c486cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c484060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c48a010 con 0x56378c487280 2024-03-09T00:13:07.871 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c39e1c0 con 0x56378c47e6a0 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56378c48a190 con 0x56378c47dd80 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bf386740 10 client.0 _ll_register_callbacks cb 0x56378c38d7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33628/0 (socket says 172.21.15.73:33628) 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 -- 172.21.15.73:0/2105642873 learned_addr learned my addr 172.21.15.73:0/2105642873 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c484060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.872 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5ba462640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c486cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 -- 172.21.15.73:0/2105642873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 msgr2=0x56378c486cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c486cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 -- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 msgr2=0x56378c484060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c484060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 -- 172.21.15.73:0/2105642873 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56378c21ef70 con 0x56378c487280 2024-03-09T00:13:07.873 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5ba462640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c486cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa5bac63640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac0056d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e6dcc4ca3487ff55 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa59dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.860+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5ac032060 con 0x56378c487280 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa5ac004a00 con 0x56378c487280 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5ac007bb0 con 0x56378c487280 2024-03-09T00:13:07.874 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5ac039050 con 0x56378c487280 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 10 client.16701 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56378c0b6170 con 0x56378c487280 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5ac007bb0 con 0x56378c487280 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa5ac02c030 con 0x56378c487280 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 10 client.16701 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 10 client.16701 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:07.875 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 20 client.16701 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c497380 0x56378c499770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 4 client.16701 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56378c3b7960 con 0x56378c497380 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.861+0000 7fa5ba462640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c497380 0x56378c499770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.862+0000 7fa59dffb640 10 client.16701 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.862+0000 7fa5ba462640 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c497380 0x56378c499770 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2a833bf871a02e3d server_cookie=50f3449f6270ce3d in_seq=0 out_seq=0 2024-03-09T00:13:07.876 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.862+0000 7fa59dffb640 1 -- 172.21.15.73:0/2105642873 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56378c3b7960 con 0x56378c497380 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.862+0000 7fa59dffb640 10 client.16701 handle_command_reply: tid=0 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.862+0000 7fa5bf386740 1 client.16701 shutdown 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 msgr2=0x56378c4847a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac007fd0 tx=0x7fa5ac0056d0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 shutdown_connections 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c497380 0x56378c499770 crc :-1 s=READY pgs=149 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c487280 0x56378c4847a0 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c484060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa59dffb640 0 client.16701 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 --2- 172.21.15.73:0/2105642873 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c486cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 >> 172.21.15.73:0/2105642873 conn(0x56378c4784b0 msgr2=0x56378c47a200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 shutdown_connections 2024-03-09T00:13:07.878 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 1 -- 172.21.15.73:0/2105642873 wait complete. 2024-03-09T00:13:07.879 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.863+0000 7fa5bf386740 20 client.16701 trim_cache size 0 max 16384 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 Processor -- start 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 -- start start 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c48e400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c48e940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41ae830 con 0x56378c47dd80 2024-03-09T00:13:07.885 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41b0df0 con 0x56378c47e6a0 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5bf386740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41aee30 con 0x56378c487280 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33638/0 (socket says 172.21.15.73:33638) 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 -- 172.21.15.73:0/363315340 learned_addr learned my addr 172.21.15.73:0/363315340 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 -- 172.21.15.73:0/363315340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 msgr2=0x56378c48e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c48e940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 -- 172.21.15.73:0/363315340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 msgr2=0x56378c48e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.886 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c48e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 -- 172.21.15.73:0/363315340 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56378c4899a0 con 0x56378c47dd80 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa5ba462640 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000bc90 tx=0x7fa5b000b6d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5d056caaadce1d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.878+0000 7fa59ffff640 1 -- 172.21.15.73:0/363315340 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0019d70 con 0x56378c47dd80 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa59ffff640 1 -- 172.21.15.73:0/363315340 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5b0019700 con 0x56378c47dd80 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa59ffff640 1 -- 172.21.15.73:0/363315340 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5b0011c80 con 0x56378c47dd80 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- 172.21.15.73:0/363315340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 msgr2=0x56378c483060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 secure :-1 s=READY pgs=844 cs=0 l=1 rev1=1 crypto rx=0x7fa5b000bc90 tx=0x7fa5b000b6d0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- 172.21.15.73:0/363315340 shutdown_connections 2024-03-09T00:13:07.887 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c487280 0x56378c48e940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c47e6a0 0x56378c48e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 --2- 172.21.15.73:0/363315340 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47dd80 0x56378c483060 unknown :-1 s=CLOSED pgs=844 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- 172.21.15.73:0/363315340 >> 172.21.15.73:0/363315340 conn(0x56378c4784b0 msgr2=0x56378c482950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- 172.21.15.73:0/363315340 shutdown_connections 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- 172.21.15.73:0/363315340 wait complete. 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 -- start start 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.879+0000 7fa5bf386740 1 Processor -- start 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c4e6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.888 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 0x56378c48d6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41af470 con 0x56378c47e6a0 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41b0790 con 0x56378c487280 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5b41afad0 con 0x56378c47dd80 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 10 client.0 _ll_register_callbacks cb 0x7fa5ac007e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33652/0 (socket says 172.21.15.73:33652) 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3781872109 learned_addr learned my addr 172.21.15.73:0/3781872109 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:07.889 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 0x56378c48d6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3781872109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 msgr2=0x56378c4e6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c4e6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 msgr2=0x56378c48d6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 0x56378c48d6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 -- 172.21.15.73:0/3781872109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56378c48ded0 con 0x56378c47e6a0 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bac63640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 0x56378c48d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5b9c61640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa5a40bce80 tx=0x7fa5a40bd010 comp rx=0 tx=0).ready entity=mon.0 client_cookie=114eaf29c461738c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:07.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa59dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5a404b040 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fa5a40b97f0 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5b0019700 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5a4004c80 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 10 client.16707 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.880+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56378c4ef270 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5b0019700 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fa5a40c5ef0 con 0x56378c47e6a0 2024-03-09T00:13:07.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 10 client.16707 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 10 client.16707 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 20 client.16707 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c4ef9a0 0x56378c4f1dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 4 client.16707 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56378c3a3080 con 0x56378c4ef9a0 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5ba462640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c4ef9a0 0x56378c4f1dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa59dffb640 10 client.16707 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:07.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.881+0000 7fa5ba462640 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c4ef9a0 0x56378c4f1dc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2c67e512006aa92 server_cookie=6ceecb782f26f0ab in_seq=0 out_seq=0 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa59dffb640 1 -- 172.21.15.73:0/3781872109 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56378c3a3080 con 0x56378c4ef9a0 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa59dffb640 10 client.16707 handle_command_reply: tid=0 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 client.16707 shutdown 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 msgr2=0x56378c4e6ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 crypto rx=0x7fa5a40bce80 tx=0x7fa5a40bd010 comp rx=0 tx=0).stop 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 shutdown_connections 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56378c4ef9a0 0x56378c4f1dc0 crc :-1 s=READY pgs=150 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56378c487280 0x56378c48d6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56378c47e6a0 0x56378c4e6ec0 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 --2- 172.21.15.73:0/3781872109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56378c47dd80 0x56378c4e6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa59dffb640 0 client.16707 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 >> 172.21.15.73:0/3781872109 conn(0x56378c4784b0 msgr2=0x56378c4831e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 shutdown_connections 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 1 -- 172.21.15.73:0/3781872109 wait complete. 2024-03-09T00:13:07.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bf386740 20 client.16707 trim_cache size 0 max 16384 2024-03-09T00:13:07.895 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:07.895 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:13:07.895 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "ddd38b63-c2e8-4d5b-87a7-8d52e399150c", 2024-03-09T00:13:07.895 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:13:07.895 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:07.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 msgr2=0x7fa5a00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 0x7fa5a00a4e60 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fa5ac010ca0 tx=0x7fa5ac015000 comp rx=0 tx=0).stop 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 msgr2=0x7fa5b41006c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.883+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 crypto rx=0x7fa5a4005940 tx=0x7fa5a402cd10 comp rx=0 tx=0).stop 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 shutdown_connections 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5b410b900 0x7fa5b4100c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5a00a29a0 0x7fa5a00a4e60 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5b4102e50 0x7fa5b41006c0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 --2- 172.21.15.73:0/3942692008 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5b4102530 0x7fa5b4100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 >> 172.21.15.73:0/3942692008 conn(0x7fa5b4076570 msgr2=0x7fa5b40fd090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 shutdown_connections 2024-03-09T00:13:07.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:07.884+0000 7fa5bc6c6640 1 -- 172.21.15.73:0/3942692008 wait complete. 2024-03-09T00:13:07.899 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:13:07.899 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:13:08.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.009+0000 7fe4224a4640 1 -- 172.21.15.73:0/2553904369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1054a0 msgr2=0x7fe41c10d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.009+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2553904369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1054a0 0x7fe41c10d070 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 crypto rx=0x7fe414009ca0 tx=0x7fe41402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 -- 172.21.15.73:0/2553904369 shutdown_connections 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2553904369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1054a0 0x7fe41c10d070 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2553904369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c100d80 0x7fe41c104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2553904369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c1003d0 0x7fe41c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 -- 172.21.15.73:0/2553904369 >> 172.21.15.73:0/2553904369 conn(0x7fe41c0772f0 msgr2=0x7fe41c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 -- 172.21.15.73:0/2553904369 shutdown_connections 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 -- 172.21.15.73:0/2553904369 wait complete. 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 Processor -- start 2024-03-09T00:13:08.008 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.010+0000 7fe4224a4640 1 -- start start 2024-03-09T00:13:08.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c1054a0 0x7fe41c198ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c199450 0x7fe41c1a3a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.010 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c110560 con 0x7fe41c1003d0 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1103e0 con 0x7fe41c199450 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1106e0 con 0x7fe41c1054a0 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33670/0 (socket says 172.21.15.73:33670) 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 -- 172.21.15.73:0/2849135411 learned_addr learned my addr 172.21.15.73:0/2849135411 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe420a41640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c199450 0x7fe41c1a3a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c1054a0 0x7fe41c198ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 -- 172.21.15.73:0/2849135411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c1054a0 msgr2=0x7fe41c198ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c1054a0 0x7fe41c198ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 -- 172.21.15.73:0/2849135411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c199450 msgr2=0x7fe41c1a3a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c199450 0x7fe41c1a3a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 -- 172.21.15.73:0/2849135411 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe41c1a41d0 con 0x7fe41c1003d0 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe41bfff640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe40c00e9f0 tx=0x7fe40c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4197fa640 1 -- 172.21.15.73:0/2849135411 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c00cd80 con 0x7fe41c1003d0 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe41c1a44c0 con 0x7fe41c1003d0 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4197fa640 1 -- 172.21.15.73:0/2849135411 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe40c010910 con 0x7fe41c1003d0 2024-03-09T00:13:08.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.011+0000 7fe4197fa640 1 -- 172.21.15.73:0/2849135411 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c0188f0 con 0x7fe41c1003d0 2024-03-09T00:13:08.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.012+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe41c1a4a10 con 0x7fe41c1003d0 2024-03-09T00:13:08.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.014+0000 7fe4197fa640 1 -- 172.21.15.73:0/2849135411 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fe40c010ac0 con 0x7fe41c1003d0 2024-03-09T00:13:08.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.014+0000 7fe4197fa640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 0x7fe4040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.014+0000 7fe425163740 1 Processor -- start 2024-03-09T00:13:08.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.014+0000 7fe425163740 1 -- start start 2024-03-09T00:13:08.019 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 0x7fe4040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe4197fa640 1 -- 172.21.15.73:0/2849135411 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe40c0cf5b0 con 0x7fe41c1003d0 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 0x55ad53f5cb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f68050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53ebfe00 con 0x55ad53f5c760 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53e68b10 con 0x55ad53f5d080 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe425163740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53ebbea0 con 0x55ad53f65c60 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 0x55ad53f5cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.020 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:44432/0 (socket says 172.21.15.73:44432) 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41b7fe640 1 -- 172.21.15.73:0/3900469646 learned_addr learned my addr 172.21.15.73:0/3900469646 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe420a41640 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f68050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 -- 172.21.15.73:0/3900469646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 msgr2=0x55ad53f68050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f68050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 0x7fe4040a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe410011550 tx=0x7fe410011950 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 -- 172.21.15.73:0/3900469646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 msgr2=0x55ad53f65720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 -- 172.21.15.73:0/3900469646 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad53c9e7d0 con 0x55ad53f5c760 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.015+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 0x55ad53f5cb40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0cc970 tx=0x7fe40c09b350 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3ac697409ccebb61 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe4027fc640 1 -- 172.21.15.73:0/3900469646 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c0a4020 con 0x55ad53f5c760 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe4027fc640 1 -- 172.21.15.73:0/3900469646 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe40c0ccbf0 con 0x55ad53f5c760 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe4027fc640 1 -- 172.21.15.73:0/3900469646 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe40c086310 con 0x55ad53f5c760 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 -- 172.21.15.73:0/3900469646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 msgr2=0x55ad53f5cb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 0x55ad53f5cb40 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 crypto rx=0x7fe40c0cc970 tx=0x7fe40c09b350 comp rx=0 tx=0).stop 2024-03-09T00:13:08.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 -- 172.21.15.73:0/3900469646 shutdown_connections 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f68050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 --2- 172.21.15.73:0/3900469646 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5c760 0x55ad53f5cb40 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 -- 172.21.15.73:0/3900469646 >> 172.21.15.73:0/3900469646 conn(0x55ad53f56e90 msgr2=0x55ad53f57270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 -- 172.21.15.73:0/3900469646 shutdown_connections 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.016+0000 7fe425163740 1 -- 172.21.15.73:0/3900469646 wait complete. 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 -- start start 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 Processor -- start 2024-03-09T00:13:08.023 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f65430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f62d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53f689f0 con 0x55ad53f5d080 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53e7c0a0 con 0x55ad53f5c760 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ad53f68b70 con 0x55ad53f65c60 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe425163740 10 client.0 _ll_register_callbacks cb 0x55ad53e7ba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.024 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33696/0 (socket says 172.21.15.73:33696) 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe41b7fe640 1 -- 172.21.15.73:0/3965657803 learned_addr learned my addr 172.21.15.73:0/3965657803 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe420a41640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f62d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.017+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f65430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 -- 172.21.15.73:0/3965657803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 msgr2=0x55ad53f62d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f62d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 -- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 msgr2=0x55ad53f65430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f65430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.025 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 -- 172.21.15.73:0/3965657803 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad53bd43f0 con 0x55ad53f5d080 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f65430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe41000faf0 tx=0x7fe410004730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c49eac480a3679 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe3fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41001a040 con 0x55ad53f5d080 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe410020d20 con 0x55ad53f5d080 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe40c0ccbf0 con 0x55ad53f5d080 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe410028400 con 0x55ad53f5d080 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe425163740 10 client.16719 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:08.026 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ad53b940f0 con 0x55ad53f5d080 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.018+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe40c0ccbf0 con 0x55ad53f5d080 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe41001d030 con 0x55ad53f5d080 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 10 client.16719 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 10 client.16719 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 20 client.16719 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53f79b30 0x55ad53f7bf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 4 client.16719 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:08.027 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ad53ecd340 con 0x55ad53f79b30 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53f79b30 0x55ad53f7bf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe3fb7fe640 10 client.16719 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.019+0000 7fe41bfff640 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53f79b30 0x55ad53f7bf20 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d85ef664a38c56a1 server_cookie=bb33915f004e6fac in_seq=0 out_seq=0 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/3965657803 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ad53ecd340 con 0x55ad53f79b30 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe3fb7fe640 10 client.16719 handle_command_reply: tid=0 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 client.16719 shutdown 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 msgr2=0x55ad53f627c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.028 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 secure :-1 s=READY pgs=849 cs=0 l=1 rev1=1 crypto rx=0x7fe41000faf0 tx=0x7fe410004730 comp rx=0 tx=0).stop 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 shutdown_connections 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53f79b30 0x55ad53f7bf20 crc :-1 s=READY pgs=151 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f65c60 0x55ad53f62d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f5d080 0x55ad53f627c0 unknown :-1 s=CLOSED pgs=849 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe3fb7fe640 0 client.16719 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.020+0000 7fe425163740 1 --2- 172.21.15.73:0/3965657803 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f65430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.021+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 >> 172.21.15.73:0/3965657803 conn(0x55ad53f56e90 msgr2=0x55ad53f58be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.021+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 shutdown_connections 2024-03-09T00:13:08.029 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.021+0000 7fe425163740 1 -- 172.21.15.73:0/3965657803 wait complete. 2024-03-09T00:13:08.030 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.021+0000 7fe425163740 20 client.16719 trim_cache size 0 max 16384 2024-03-09T00:13:08.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 Processor -- start 2024-03-09T00:13:08.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 -- start start 2024-03-09T00:13:08.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5c760 0x55ad53f64ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.035 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53f6cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1aa4c0 con 0x55ad53f65c60 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1aca80 con 0x55ad53f5d080 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe425163740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1aaac0 con 0x55ad53f5c760 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe420a41640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53f6cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe41bfff640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5c760 0x55ad53f64ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:44460/0 (socket says 172.21.15.73:44460) 2024-03-09T00:13:08.036 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe41b7fe640 1 -- 172.21.15.73:0/1994242056 learned_addr learned my addr 172.21.15.73:0/1994242056 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.038+0000 7fe41b7fe640 1 -- 172.21.15.73:0/1994242056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5c760 msgr2=0x55ad53f64ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5c760 0x55ad53f64ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe41b7fe640 1 -- 172.21.15.73:0/1994242056 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 msgr2=0x55ad53f6cdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53f6cdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe41b7fe640 1 -- 172.21.15.73:0/1994242056 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ad53f5ceb0 con 0x55ad53f5d080 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe420a41640 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53f6cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:13:08.042 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe41b7fe640 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4100097a0 tx=0x7fe410005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=382498ac74b52440 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe401ffb640 1 -- 172.21.15.73:0/1994242056 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41003b070 con 0x55ad53f5d080 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe401ffb640 1 -- 172.21.15.73:0/1994242056 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe41001f3b0 con 0x55ad53f5d080 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.039+0000 7fe401ffb640 1 -- 172.21.15.73:0/1994242056 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe41001a040 con 0x55ad53f5d080 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- 172.21.15.73:0/1994242056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 msgr2=0x55ad53f65430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fe4100097a0 tx=0x7fe410005d30 comp rx=0 tx=0).stop 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- 172.21.15.73:0/1994242056 shutdown_connections 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53f6cdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.043 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5d080 0x55ad53f65430 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 --2- 172.21.15.73:0/1994242056 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5c760 0x55ad53f64ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- 172.21.15.73:0/1994242056 >> 172.21.15.73:0/1994242056 conn(0x55ad53f56e90 msgr2=0x55ad53f62780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- 172.21.15.73:0/1994242056 shutdown_connections 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- 172.21.15.73:0/1994242056 wait complete. 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 -- start start 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.040+0000 7fe425163740 1 Processor -- start 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f6b080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5d080 0x55ad53f6b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.044 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1ab100 con 0x55ad53f65c60 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1ac420 con 0x55ad53f5c760 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe41c1ab760 con 0x55ad53f5d080 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe425163740 10 client.0 _ll_register_callbacks cb 0x7fe410009d10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe41b7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5d080 0x55ad53f6b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33722/0 (socket says 172.21.15.73:33722) 2024-03-09T00:13:08.045 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 -- 172.21.15.73:0/1985747342 learned_addr learned my addr 172.21.15.73:0/1985747342 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe41bfff640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f6b080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 -- 172.21.15.73:0/1985747342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5d080 msgr2=0x55ad53f6b5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5d080 0x55ad53f6b5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 -- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 msgr2=0x55ad53f6b080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f6b080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 -- 172.21.15.73:0/1985747342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ad53fc96f0 con 0x55ad53f65c60 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe420a41640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe414006010 tx=0x7fe414032df0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=66c540ee25404075 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.041+0000 7fe3fb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:08.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe414039040 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 9437+0+0 (secure 0 0 0) 0x7fe4140329a0 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe41001f3b0 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe414054050 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 10 client.16725 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ad53fd0a70 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe41001f3b0 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fe41404b480 con 0x55ad53f65c60 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 10 client.16725 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:08.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 10 client.16725 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 20 client.16725 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53fd11a0 0x55ad53fd35c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 4 client.16725 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ad53ec7690 con 0x55ad53fd11a0 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.042+0000 7fe41bfff640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53fd11a0 0x55ad53fd35c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe3fb7fe640 10 client.16725 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe41bfff640 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53fd11a0 0x55ad53fd35c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5b0d575625b49f7 server_cookie=f0ccb49b5b9b9096 in_seq=0 out_seq=0 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe3fb7fe640 1 -- 172.21.15.73:0/1985747342 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55ad53ec7690 con 0x55ad53fd11a0 2024-03-09T00:13:08.048 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe3fb7fe640 10 client.16725 handle_command_reply: tid=0 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe425163740 1 client.16725 shutdown 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 msgr2=0x55ad53fc90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.043+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 crypto rx=0x7fe414006010 tx=0x7fe414032df0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 shutdown_connections 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ad53fd11a0 0x55ad53fd35c0 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ad53f65c60 0x55ad53fc90b0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ad53f5d080 0x55ad53f6b5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe3fb7fe640 0 client.16725 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:08.049 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 --2- 172.21.15.73:0/1985747342 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ad53f5c760 0x55ad53f6b080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 >> 172.21.15.73:0/1985747342 conn(0x55ad53f56e90 msgr2=0x55ad53f60fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 shutdown_connections 2024-03-09T00:13:08.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 1 -- 172.21.15.73:0/1985747342 wait complete. 2024-03-09T00:13:08.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe425163740 20 client.16725 trim_cache size 0 max 16384 2024-03-09T00:13:08.050 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2479 inodes in the stack)", 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "ddd38b63-c2e8-4d5b-87a7-8d52e399150c": { 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "ddd38b63-c2e8-4d5b-87a7-8d52e399150c", 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:13:08.051 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:13:08.052 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:13:08.052 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:08.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 msgr2=0x7fe4040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 0x7fe4040a4e30 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fe410011550 tx=0x7fe410011950 comp rx=0 tx=0).stop 2024-03-09T00:13:08.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 msgr2=0x7fe41c198980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:08.053 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 secure :-1 s=READY pgs=847 cs=0 l=1 rev1=1 crypto rx=0x7fe40c00e9f0 tx=0x7fe40c00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 shutdown_connections 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fe41c199450 0x7fe41c1a3a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fe41c1054a0 0x7fe41c198ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fe4040a2970 0x7fe4040a4e30 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 --2- 172.21.15.73:0/2849135411 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fe41c1003d0 0x7fe41c198980 unknown :-1 s=CLOSED pgs=847 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 >> 172.21.15.73:0/2849135411 conn(0x7fe41c0772f0 msgr2=0x7fe41c0fe320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:08.054 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 shutdown_connections 2024-03-09T00:13:08.055 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:08.044+0000 7fe4224a4640 1 -- 172.21.15.73:0/2849135411 wait complete. 2024-03-09T00:13:08.055 INFO:tasks.cephfs.filesystem:scrub status for tag:ddd38b63-c2e8-4d5b-87a7-8d52e399150c - {'path': '/', 'tag': 'ddd38b63-c2e8-4d5b-87a7-8d52e399150c', 'options': 'recursive,force'} 2024-03-09T00:13:08.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:08 smithi073 ceph-mon[32309]: pgmap v191: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:08.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:08 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:13:08.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:08 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:08.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:08 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:13:08.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:08 smithi102 ceph-mon[36106]: pgmap v191: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:08.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:08 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:13:08.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:08 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:08.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:08 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:13:08.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:08 smithi179 ceph-mon[36637]: pgmap v191: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 45 KiB/s wr, 19 op/s 2024-03-09T00:13:08.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:08 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:13:08.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:08 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:08.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:08 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:13:09.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:09 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:09.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:09 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:13:09.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:09 smithi102 ceph-mon[36106]: pgmap v192: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:09.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:09 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:09.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:09 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:13:09.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:09 smithi179 ceph-mon[36637]: pgmap v192: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:09.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:09 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:09.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:09 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:13:09.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:09 smithi073 ceph-mon[32309]: pgmap v192: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 45 KiB/s wr, 18 op/s 2024-03-09T00:13:10.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:10 smithi102 ceph-mon[36106]: osdmap e95: 12 total, 12 up, 12 in 2024-03-09T00:13:10.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:10 smithi179 ceph-mon[36637]: osdmap e95: 12 total, 12 up, 12 in 2024-03-09T00:13:10.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:10 smithi073 ceph-mon[32309]: osdmap e95: 12 total, 12 up, 12 in 2024-03-09T00:13:11.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:11 smithi102 ceph-mon[36106]: pgmap v194: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-09T00:13:11.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:11 smithi179 ceph-mon[36637]: pgmap v194: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-09T00:13:11.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:11 smithi073 ceph-mon[32309]: pgmap v194: 129 pgs: 129 active+clean; 35 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 54 KiB/s wr, 22 op/s 2024-03-09T00:13:14.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:14 smithi073 ceph-mon[32309]: pgmap v195: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:14.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:14 smithi102 ceph-mon[36106]: pgmap v195: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:14.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:14 smithi179 ceph-mon[36637]: pgmap v195: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:16.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:15 smithi102 ceph-mon[36106]: osdmap e96: 12 total, 12 up, 12 in 2024-03-09T00:13:16.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:15 smithi102 ceph-mon[36106]: pgmap v197: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 127 B/s wr, 431 op/s 2024-03-09T00:13:16.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:15 smithi179 ceph-mon[36637]: osdmap e96: 12 total, 12 up, 12 in 2024-03-09T00:13:16.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:15 smithi179 ceph-mon[36637]: pgmap v197: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 127 B/s wr, 431 op/s 2024-03-09T00:13:16.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:15 smithi073 ceph-mon[32309]: osdmap e96: 12 total, 12 up, 12 in 2024-03-09T00:13:16.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:15 smithi073 ceph-mon[32309]: pgmap v197: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 431 KiB/s rd, 127 B/s wr, 431 op/s 2024-03-09T00:13:18.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:18 smithi073 ceph-mon[32309]: pgmap v198: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 255 B/s wr, 432 op/s 2024-03-09T00:13:18.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:18 smithi102 ceph-mon[36106]: pgmap v198: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 255 B/s wr, 432 op/s 2024-03-09T00:13:18.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:18 smithi179 ceph-mon[36637]: pgmap v198: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 432 KiB/s rd, 255 B/s wr, 432 op/s 2024-03-09T00:13:20.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:20 smithi073 ceph-mon[32309]: pgmap v199: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 206 B/s wr, 349 op/s 2024-03-09T00:13:20.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:20 smithi102 ceph-mon[36106]: pgmap v199: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 206 B/s wr, 349 op/s 2024-03-09T00:13:20.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:20 smithi179 ceph-mon[36637]: pgmap v199: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 349 KiB/s rd, 206 B/s wr, 349 op/s 2024-03-09T00:13:22.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:22 smithi073 ceph-mon[32309]: pgmap v200: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:22.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:22 smithi102 ceph-mon[36106]: pgmap v200: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:22.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:22 smithi179 ceph-mon[36637]: pgmap v200: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:13:24.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:24 smithi073 ceph-mon[32309]: pgmap v201: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:13:24.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:24 smithi102 ceph-mon[36106]: pgmap v201: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:13:24.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:24 smithi179 ceph-mon[36637]: pgmap v201: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:13:26.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:26 smithi073 ceph-mon[32309]: pgmap v202: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:13:26.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:26 smithi102 ceph-mon[36106]: pgmap v202: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:13:26.583 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:26 smithi179 ceph-mon[36637]: pgmap v202: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:13:28.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:28 smithi073 ceph-mon[32309]: pgmap v203: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:13:28.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:28 smithi102 ceph-mon[36106]: pgmap v203: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:13:28.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:28 smithi179 ceph-mon[36637]: pgmap v203: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:13:30.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:30 smithi073 ceph-mon[32309]: pgmap v204: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:30.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:30 smithi102 ceph-mon[36106]: pgmap v204: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:30.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:30 smithi179 ceph-mon[36637]: pgmap v204: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:32.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:32 smithi073 ceph-mon[32309]: pgmap v205: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:32.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:32 smithi102 ceph-mon[36106]: pgmap v205: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:32.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:32 smithi179 ceph-mon[36637]: pgmap v205: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:34.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:34 smithi073 ceph-mon[32309]: pgmap v206: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:34.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:34 smithi102 ceph-mon[36106]: pgmap v206: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:34.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:34 smithi179 ceph-mon[36637]: pgmap v206: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:36.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:36 smithi102 ceph-mon[36106]: pgmap v207: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:36.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:36 smithi179 ceph-mon[36637]: pgmap v207: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:36.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:36 smithi073 ceph-mon[32309]: pgmap v207: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 1 op/s 2024-03-09T00:13:38.056 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:13:38.056 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 -- 172.21.15.73:0/2549220369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08100dc0 msgr2=0x7fee08104e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 --2- 172.21.15.73:0/2549220369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08100dc0 0x7fee08104e00 secure :-1 s=READY pgs=851 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009900 tx=0x7fedf002bc30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 -- 172.21.15.73:0/2549220369 shutdown_connections 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 --2- 172.21.15.73:0/2549220369 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 0x7fee0810d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 --2- 172.21.15.73:0/2549220369 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08100dc0 0x7fee08104e00 unknown :-1 s=CLOSED pgs=851 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.174 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 --2- 172.21.15.73:0/2549220369 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 0x7fee081007f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 -- 172.21.15.73:0/2549220369 >> 172.21.15.73:0/2549220369 conn(0x7fee08076570 msgr2=0x7fee08076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 -- 172.21.15.73:0/2549220369 shutdown_connections 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.174+0000 7fee0ce04640 1 -- 172.21.15.73:0/2549220369 wait complete. 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 Processor -- start 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 -- start start 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 0x7fee08198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 0x7fee08198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee08110570 con 0x7fee08199500 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081103f0 con 0x7fee08100410 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee0ce04640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081106f0 con 0x7fee081054e0 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee06d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee06d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45332/0 (socket says 172.21.15.73:45332) 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fee06d9d640 1 -- 172.21.15.73:0/1472984772 learned_addr learned my addr 172.21.15.73:0/1472984772 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.175+0000 7fedfffff640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 0x7fee08198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee0659c640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 0x7fee08198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 -- 172.21.15.73:0/1472984772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 msgr2=0x7fee08198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 0x7fee08198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 -- 172.21.15.73:0/1472984772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 msgr2=0x7fee08198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 0x7fee08198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 -- 172.21.15.73:0/1472984772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee081a4280 con 0x7fee08199500 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee06d9d640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee0000eb10 tx=0x7fee0000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fedff7fe640 1 -- 172.21.15.73:0/1472984772 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00004b20 con 0x7fee08199500 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee081a4570 con 0x7fee08199500 2024-03-09T00:13:38.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fedff7fe640 1 -- 172.21.15.73:0/1472984772 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee00004cc0 con 0x7fee08199500 2024-03-09T00:13:38.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fedff7fe640 1 -- 172.21.15.73:0/1472984772 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee00010760 con 0x7fee08199500 2024-03-09T00:13:38.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.176+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee081a4ae0 con 0x7fee08199500 2024-03-09T00:13:38.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fedff7fe640 1 -- 172.21.15.73:0/1472984772 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fee00005020 con 0x7fee08199500 2024-03-09T00:13:38.188 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 Processor -- start 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 -- start start 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f6eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8ed1e00 con 0x55cfe8f77c60 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8e7ab10 con 0x55cfe8f6f080 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8ecdea0 con 0x55cfe8f6e760 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.179+0000 7fedff7fe640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 0x7fedf40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.189 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45346/0 (socket says 172.21.15.73:45346) 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 -- 172.21.15.73:0/2667443766 learned_addr learned my addr 172.21.15.73:0/2667443766 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fee06d9d640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 0x7fedf40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f6eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fee0659c640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedff7fe640 1 -- 172.21.15.73:0/1472984772 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fee000cfa20 con 0x7fee08199500 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 -- 172.21.15.73:0/2667443766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 msgr2=0x55cfe8f6eb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f6eb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 -- 172.21.15.73:0/2667443766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 msgr2=0x55cfe8f77720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 -- 172.21.15.73:0/2667443766 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfe8cb07d0 con 0x55cfe8f77c60 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fee0659c640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fee06d9d640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 0x7fedf40a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee0009b990 tx=0x7fee0000efd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfffff640 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005c40 tx=0x7fedf00058b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=326a12afda3a2de3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfcff9640 1 -- 172.21.15.73:0/2667443766 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0049070 con 0x55cfe8f77c60 2024-03-09T00:13:38.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfcff9640 1 -- 172.21.15.73:0/2667443766 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00035f0 con 0x55cfe8f77c60 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.180+0000 7fedfcff9640 1 -- 172.21.15.73:0/2667443766 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf00426a0 con 0x55cfe8f77c60 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 -- 172.21.15.73:0/2667443766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 msgr2=0x55cfe8f7a050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 crypto rx=0x7fedf0005c40 tx=0x7fedf00058b0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 -- 172.21.15.73:0/2667443766 shutdown_connections 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7a050 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/2667443766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f6eb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.192 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 -- 172.21.15.73:0/2667443766 >> 172.21.15.73:0/2667443766 conn(0x55cfe8f68e90 msgr2=0x55cfe8f69270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 -- 172.21.15.73:0/2667443766 shutdown_connections 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.181+0000 7fee0f9e9740 1 -- 172.21.15.73:0/2667443766 wait complete. 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 -- start start 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 Processor -- start 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f77450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 0x55cfe8f74d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8f7a9f0 con 0x55cfe8f6f080 2024-03-09T00:13:38.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8e8e0a0 con 0x55cfe8f77c60 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cfe8f7ab70 con 0x55cfe8f6e760 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0f9e9740 10 client.0 _ll_register_callbacks cb 0x55cfe8e8da70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45358/0 (socket says 172.21.15.73:45358) 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 -- 172.21.15.73:0/4201177164 learned_addr learned my addr 172.21.15.73:0/4201177164 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fedfffff640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f77450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fedfffff640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 0x55cfe8f74d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 -- 172.21.15.73:0/4201177164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 msgr2=0x55cfe8f77450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f77450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 -- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 msgr2=0x55cfe8f74d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 0x55cfe8f74d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 -- 172.21.15.73:0/4201177164 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfe8be63f0 con 0x55cfe8f6f080 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fedfffff640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 0x55cfe8f74d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fee0659c640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fedf8010670 tx=0x7fedf8010b30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c1d84f6c392e908b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.182+0000 7fededffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf800ead0 con 0x55cfe8f6f080 2024-03-09T00:13:38.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fedf800ec70 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedf00035f0 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf8010e70 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 10 client.16737 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cfe8ba60f0 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf00035f0 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fedf800f040 con 0x55cfe8f6f080 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 10 client.16737 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:38.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 10 client.16737 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 20 client.16737 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8f87780 0x55cfe8f89b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 4 client.16737 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.183+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cfe8edf340 con 0x55cfe8f87780 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.184+0000 7fedfffff640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8f87780 0x55cfe8f89b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.184+0000 7fededffb640 10 client.16737 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.184+0000 7fedfffff640 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8f87780 0x55cfe8f89b70 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dff6f3ab1c16a84c server_cookie=c922dc01faabfd62 in_seq=0 out_seq=0 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fededffb640 1 -- 172.21.15.73:0/4201177164 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cfe8edf340 con 0x55cfe8f87780 2024-03-09T00:13:38.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fededffb640 10 client.16737 handle_command_reply: tid=0 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 client.16737 shutdown 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 msgr2=0x55cfe8f747e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 secure :-1 s=READY pgs=854 cs=0 l=1 rev1=1 crypto rx=0x7fedf8010670 tx=0x7fedf8010b30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 shutdown_connections 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8f87780 0x55cfe8f89b70 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f77c60 0x55cfe8f74d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f6f080 0x55cfe8f747e0 unknown :-1 s=CLOSED pgs=854 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fededffb640 0 client.16737 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/4201177164 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f77450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 >> 172.21.15.73:0/4201177164 conn(0x55cfe8f68e90 msgr2=0x55cfe8f6abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 shutdown_connections 2024-03-09T00:13:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 1 -- 172.21.15.73:0/4201177164 wait complete. 2024-03-09T00:13:38.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.185+0000 7fee0f9e9740 20 client.16737 trim_cache size 0 max 16384 2024-03-09T00:13:38.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 Processor -- start 2024-03-09T00:13:38.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 -- start start 2024-03-09T00:13:38.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f76f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081ac8a0 con 0x55cfe8f77c60 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081aee40 con 0x55cfe8f6f080 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081acea0 con 0x55cfe8f6e760 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45366/0 (socket says 172.21.15.73:45366) 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 -- 172.21.15.73:0/463210971 learned_addr learned my addr 172.21.15.73:0/463210971 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f76f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 -- 172.21.15.73:0/463210971 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 msgr2=0x55cfe8f76f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f76f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 -- 172.21.15.73:0/463210971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 msgr2=0x55cfe8f77450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 -- 172.21.15.73:0/463210971 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cfe8f6ee70 con 0x55cfe8f77c60 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.200+0000 7fedfffff640 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009900 tx=0x7fedf0000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=399fb293608bd78b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fedeffff640 1 -- 172.21.15.73:0/463210971 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0056070 con 0x55cfe8f77c60 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fedeffff640 1 -- 172.21.15.73:0/463210971 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf0004440 con 0x55cfe8f77c60 2024-03-09T00:13:38.207 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fedeffff640 1 -- 172.21.15.73:0/463210971 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0049040 con 0x55cfe8f77c60 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- 172.21.15.73:0/463210971 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 msgr2=0x55cfe8f7edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009900 tx=0x7fedf0000bc0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- 172.21.15.73:0/463210971 shutdown_connections 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7edb0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f77450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/463210971 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f76f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- 172.21.15.73:0/463210971 >> 172.21.15.73:0/463210971 conn(0x55cfe8f68e90 msgr2=0x55cfe8f8e510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- 172.21.15.73:0/463210971 shutdown_connections 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- 172.21.15.73:0/463210971 wait complete. 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 -- start start 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.201+0000 7fee0f9e9740 1 Processor -- start 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f7da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f7df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081ad4c0 con 0x55cfe8f77c60 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081ae7e0 con 0x55cfe8f6f080 2024-03-09T00:13:38.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee081adb20 con 0x55cfe8f6e760 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 10 client.0 _ll_register_callbacks cb 0x7fedf80093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45372/0 (socket says 172.21.15.73:45372) 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 -- 172.21.15.73:0/3076709160 learned_addr learned my addr 172.21.15.73:0/3076709160 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0659c640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f7df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f7da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 -- 172.21.15.73:0/3076709160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 msgr2=0x55cfe8f7da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f7da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 -- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 msgr2=0x55cfe8f7df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f7df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 -- 172.21.15.73:0/3076709160 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cfe8fddec0 con 0x55cfe8f77c60 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fedfffff640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009690 tx=0x7fedf00035f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fceefe0769171a40 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fededffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fedf0049040 con 0x55cfe8f77c60 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fedf00517a0 con 0x55cfe8f77c60 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fedf0004440 con 0x55cfe8f77c60 2024-03-09T00:13:38.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.202+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fedf0056040 con 0x55cfe8f77c60 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 10 client.16743 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cfe8fdf1d0 con 0x55cfe8f77c60 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fedf0004440 con 0x55cfe8f77c60 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fedf004f040 con 0x55cfe8f77c60 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 10 client.16743 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 10 client.16743 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 20 client.16743 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8fdf400 0x55cfe8fe17f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 4 client.16743 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55cfe8ed9690 con 0x55cfe8fdf400 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0659c640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8fdf400 0x55cfe8fe17f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fededffb640 10 client.16743 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.203+0000 7fee0659c640 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8fdf400 0x55cfe8fe17f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c7bbfd196ede933d server_cookie=b37c612187aafd7b in_seq=0 out_seq=0 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fededffb640 1 -- 172.21.15.73:0/3076709160 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55cfe8ed9690 con 0x55cfe8fdf400 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fededffb640 10 client.16743 handle_command_reply: tid=0 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 client.16743 shutdown 2024-03-09T00:13:38.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 msgr2=0x55cfe8f7e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 crypto rx=0x7fedf0009690 tx=0x7fedf00035f0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 shutdown_connections 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cfe8fdf400 0x55cfe8fe17f0 crc :-1 s=READY pgs=154 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cfe8f77c60 0x55cfe8f7e4c0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cfe8f6f080 0x55cfe8f7df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 --2- 172.21.15.73:0/3076709160 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cfe8f6e760 0x55cfe8f7da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fededffb640 0 client.16743 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 >> 172.21.15.73:0/3076709160 conn(0x55cfe8f68e90 msgr2=0x55cfe8f74780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 shutdown_connections 2024-03-09T00:13:38.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 1 -- 172.21.15.73:0/3076709160 wait complete. 2024-03-09T00:13:38.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0f9e9740 20 client.16743 trim_cache size 0 max 16384 2024-03-09T00:13:38.215 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:38.215 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:13:38.215 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:13:38.216 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:38.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 msgr2=0x7fedf40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 0x7fedf40a4e30 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fee0009b990 tx=0x7fee0000efd0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 msgr2=0x7fee081a3ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 crypto rx=0x7fee0000eb10 tx=0x7fee0000c2d0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 shutdown_connections 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fedf40a2970 0x7fedf40a4e30 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fee08199500 0x7fee081a3ab0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fee081054e0 0x7fee08198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 --2- 172.21.15.73:0/1472984772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fee08100410 0x7fee08198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.204+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 >> 172.21.15.73:0/1472984772 conn(0x7fee08076570 msgr2=0x7fee080fe0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.205+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 shutdown_connections 2024-03-09T00:13:38.218 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.205+0000 7fee0ce04640 1 -- 172.21.15.73:0/1472984772 wait complete. 2024-03-09T00:13:38.219 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:13:38.219 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:13:38.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 -- 172.21.15.73:0/4154789121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100e10 msgr2=0x7f58b8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 --2- 172.21.15.73:0/4154789121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100e10 0x7f58b8104e50 secure :-1 s=READY pgs=857 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009900 tx=0x7f58ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 -- 172.21.15.73:0/4154789121 shutdown_connections 2024-03-09T00:13:38.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 --2- 172.21.15.73:0/4154789121 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 0x7f58b810d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 --2- 172.21.15.73:0/4154789121 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100e10 0x7f58b8104e50 unknown :-1 s=CLOSED pgs=857 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 --2- 172.21.15.73:0/4154789121 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8100460 0x7f58b8100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.323+0000 7f58bf52b640 1 -- 172.21.15.73:0/4154789121 >> 172.21.15.73:0/4154789121 conn(0x7f58b80772f0 msgr2=0x7f58b8077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- 172.21.15.73:0/4154789121 shutdown_connections 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- 172.21.15.73:0/4154789121 wait complete. 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 Processor -- start 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- start start 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 0x7f58b819ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8197ab0 0x7f58b8197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b81105a0 con 0x7f58b8100460 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8110420 con 0x7f58b8105530 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bf52b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8110720 con 0x7f58b8197ab0 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bd2c7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bd2c7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:45378/0 (socket says 172.21.15.73:45378) 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bd2c7640 1 -- 172.21.15.73:0/1357420176 learned_addr learned my addr 172.21.15.73:0/1357420176 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.324+0000 7f58bcac6640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 0x7f58b819ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bdac8640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8197ab0 0x7f58b8197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 -- 172.21.15.73:0/1357420176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8197ab0 msgr2=0x7f58b8197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8197ab0 0x7f58b8197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 -- 172.21.15.73:0/1357420176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 msgr2=0x7f58b819ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 0x7f58b819ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 -- 172.21.15.73:0/1357420176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b81985d0 con 0x7f58b8100460 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bd2c7640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c890 tx=0x7f58a800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a8004220 con 0x7f58b8100460 2024-03-09T00:13:38.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f58a80043c0 con 0x7f58b8100460 2024-03-09T00:13:38.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f58a80049e0 con 0x7f58b8100460 2024-03-09T00:13:38.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b81988c0 con 0x7f58b8100460 2024-03-09T00:13:38.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.325+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b81a4ac0 con 0x7f58b8100460 2024-03-09T00:13:38.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.326+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b80680e0 con 0x7f58b8100460 2024-03-09T00:13:38.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.327+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f58a80079b0 con 0x7f58b8100460 2024-03-09T00:13:38.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.327+0000 7f58b67fc640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 0x7f58a00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.328+0000 7f58bcac6640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 0x7f58a00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.328+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f58a80cf300 con 0x7f58b8100460 2024-03-09T00:13:38.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.328+0000 7f58bcac6640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 0x7f58a00a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009230 tx=0x7f58ac005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.331+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f58a8014030 con 0x7f58b8100460 2024-03-09T00:13:38.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:38 smithi073 ceph-mon[32309]: pgmap v208: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:38.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:38 smithi102 ceph-mon[36106]: pgmap v208: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:38.538 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.541+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f58b8198f90 con 0x7f58b8100460 2024-03-09T00:13:38.539 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.542+0000 7f58b67fc640 1 -- 172.21.15.73:0/1357420176 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f58a809bc70 con 0x7f58b8100460 2024-03-09T00:13:38.539 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:13:38.539 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:13:38.540 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:13:38.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 msgr2=0x7f58a00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 0x7f58a00a4e30 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f58ac009230 tx=0x7f58ac005fd0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 msgr2=0x7f58b819d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 crypto rx=0x7f58a800c890 tx=0x7f58a800cd50 comp rx=0 tx=0).stop 2024-03-09T00:13:38.541 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 shutdown_connections 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f58b8197ab0 0x7f58b8197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f58b8105530 0x7f58b819ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f58a00a2970 0x7f58a00a4e30 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 --2- 172.21.15.73:0/1357420176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f58b8100460 0x7f58b819d8b0 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.543+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 >> 172.21.15.73:0/1357420176 conn(0x7f58b80772f0 msgr2=0x7f58b8104210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.544+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 shutdown_connections 2024-03-09T00:13:38.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.544+0000 7f58bf52b640 1 -- 172.21.15.73:0/1357420176 wait complete. 2024-03-09T00:13:38.551 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:13:38.552 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:13:38.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:38 smithi179 ceph-mon[36637]: pgmap v208: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:38.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- 172.21.15.73:0/2386370655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81003e0 msgr2=0x7f22b81007c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 --2- 172.21.15.73:0/2386370655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81003e0 0x7f22b81007c0 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009900 tx=0x7f22ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- 172.21.15.73:0/2386370655 shutdown_connections 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 --2- 172.21.15.73:0/2386370655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22b81054b0 0x7f22b810d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 --2- 172.21.15.73:0/2386370655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b8100d90 0x7f22b8104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 --2- 172.21.15.73:0/2386370655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81003e0 0x7f22b81007c0 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- 172.21.15.73:0/2386370655 >> 172.21.15.73:0/2386370655 conn(0x7f22b80773a0 msgr2=0x7f22b80777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- 172.21.15.73:0/2386370655 shutdown_connections 2024-03-09T00:13:38.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- 172.21.15.73:0/2386370655 wait complete. 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 Processor -- start 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.658+0000 7f22c0043640 1 -- start start 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b81003e0 0x7f22b819d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22b81979e0 0x7f22b8197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8069100 con 0x7f22b81054b0 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8068f80 con 0x7f22b81003e0 2024-03-09T00:13:38.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b8069280 con 0x7f22b81979e0 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46672/0 (socket says 172.21.15.73:46672) 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 -- 172.21.15.73:0/2966172091 learned_addr learned my addr 172.21.15.73:0/2966172091 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bdddf640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b81003e0 0x7f22b819d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 -- 172.21.15.73:0/2966172091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22b81979e0 msgr2=0x7f22b8197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22b81979e0 0x7f22b8197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 -- 172.21.15.73:0/2966172091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b81003e0 msgr2=0x7f22b819d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b81003e0 0x7f22b819d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.660 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 -- 172.21.15.73:0/2966172091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22b8198500 con 0x7f22b81054b0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22bd5de640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c970 tx=0x7f22b400ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22aaffd640 1 -- 172.21.15.73:0/2966172091 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b4007bb0 con 0x7f22b81054b0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22aaffd640 1 -- 172.21.15.73:0/2966172091 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22b4007d50 con 0x7f22b81054b0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22b81987f0 con 0x7f22b81054b0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22aaffd640 1 -- 172.21.15.73:0/2966172091 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b400f450 con 0x7f22b81054b0 2024-03-09T00:13:38.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.659+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22b81a49f0 con 0x7f22b81054b0 2024-03-09T00:13:38.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.661+0000 7f22aaffd640 1 -- 172.21.15.73:0/2966172091 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f22b40046c0 con 0x7f22b81054b0 2024-03-09T00:13:38.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 Processor -- start 2024-03-09T00:13:38.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 -- start start 2024-03-09T00:13:38.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22aaffd640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 0x7f22a40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b15f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b161a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b11d360 con 0x55d79b1561a0 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b0b9ec0 con 0x55d79b156ac0 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b062d30 con 0x55d79b15f6a0 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22be5e0640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 0x7f22a40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46686/0 (socket says 172.21.15.73:46686) 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22bdddf640 1 -- 172.21.15.73:0/1096755060 learned_addr learned my addr 172.21.15.73:0/1096755060 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22aaffd640 1 -- 172.21.15.73:0/2966172091 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f22b40cf280 con 0x7f22b81054b0 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22be5e0640 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b161a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.662+0000 7f22bd5de640 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b15f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 -- 172.21.15.73:0/1096755060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 msgr2=0x55d79b161a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b161a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 -- 172.21.15.73:0/1096755060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 msgr2=0x55d79b15f160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b15f160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 -- 172.21.15.73:0/1096755060 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d79b061bc0 con 0x55d79b1561a0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22be5e0640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 0x7f22a40a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f22b000af30 tx=0x7f22b000e7e0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22bdddf640 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009230 tx=0x7f22ac02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67073c7a7c0bdcd6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22a3fff640 1 -- 172.21.15.73:0/1096755060 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac042070 con 0x55d79b1561a0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22a3fff640 1 -- 172.21.15.73:0/1096755060 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22ac02cb00 con 0x55d79b1561a0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.663+0000 7f22a3fff640 1 -- 172.21.15.73:0/1096755060 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac0051a0 con 0x55d79b1561a0 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- 172.21.15.73:0/1096755060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 msgr2=0x55d79b156580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 crypto rx=0x7f22ac009230 tx=0x7f22ac02c650 comp rx=0 tx=0).stop 2024-03-09T00:13:38.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- 172.21.15.73:0/1096755060 shutdown_connections 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b161a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b15f160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/1096755060 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b156580 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- 172.21.15.73:0/1096755060 >> 172.21.15.73:0/1096755060 conn(0x55d79b1508d0 msgr2=0x55d79b150cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- 172.21.15.73:0/1096755060 shutdown_connections 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- 172.21.15.73:0/1096755060 wait complete. 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- start start 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 Processor -- start 2024-03-09T00:13:38.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b1561a0 0x55d79b15c680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b0584a0 con 0x55d79b15f6a0 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b0b5f60 con 0x55d79b1561a0 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.664+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d79b162430 con 0x55d79b156ac0 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22c2c1c740 10 client.0 _ll_register_callbacks cb 0x55d79b0b3d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46700/0 (socket says 172.21.15.73:46700) 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 -- 172.21.15.73:0/228375280 learned_addr learned my addr 172.21.15.73:0/228375280 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 -- 172.21.15.73:0/228375280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 msgr2=0x55d79b15cbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 -- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b1561a0 msgr2=0x55d79b15c680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22bdddf640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b1561a0 0x55d79b15c680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b1561a0 0x55d79b15c680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 -- 172.21.15.73:0/228375280 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d79adce3f0 con 0x55d79b15f6a0 2024-03-09T00:13:38.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22be5e0640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f22b001bcc0 tx=0x7f22b0004c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=32b016d6429714cc server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22b0018c80 con 0x55d79b15f6a0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f22b0027480 con 0x55d79b15f6a0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22ac02cb00 con 0x55d79b15f6a0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22b001be70 con 0x55d79b15f6a0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22c2c1c740 10 client.16761 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d79ae987d0 con 0x55d79b15f6a0 2024-03-09T00:13:38.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.665+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22ac02cb00 con 0x55d79b15f6a0 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f22b001a070 con 0x55d79b15f6a0 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 10 client.16761 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 10 client.16761 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 20 client.16761 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b16f850 0x55d79b171ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 4 client.16761 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d79b0c7400 con 0x55d79b16f850 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22bdddf640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b16f850 0x55d79b171ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22bdddf640 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b16f850 0x55d79b171ca0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5cfa86c9136578b8 server_cookie=3f20e1d7131352d9 in_seq=0 out_seq=0 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.666+0000 7f22a17fa640 10 client.16761 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22a17fa640 1 -- 172.21.15.73:0/228375280 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d79b0c7400 con 0x55d79b16f850 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22a17fa640 10 client.16761 handle_command_reply: tid=0 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 client.16761 shutdown 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 msgr2=0x55d79b15d100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 crypto rx=0x7f22b001bcc0 tx=0x7f22b0004c10 comp rx=0 tx=0).stop 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 shutdown_connections 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b16f850 0x55d79b171ca0 crc :-1 s=READY pgs=155 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b15f6a0 0x55d79b15d100 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15cbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/228375280 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b1561a0 0x55d79b15c680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22a17fa640 0 client.16761 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 >> 172.21.15.73:0/228375280 conn(0x55d79b1508d0 msgr2=0x55d79b152620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 shutdown_connections 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 1 -- 172.21.15.73:0/228375280 wait complete. 2024-03-09T00:13:38.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.667+0000 7f22c2c1c740 20 client.16761 trim_cache size 0 max 16384 2024-03-09T00:13:38.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 Processor -- start 2024-03-09T00:13:38.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 -- start start 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b15f6a0 0x55d79b15b490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81ac870 con 0x55d79b1561a0 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81aee10 con 0x55d79b15f6a0 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81ace70 con 0x55d79b156ac0 2024-03-09T00:13:38.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46712/0 (socket says 172.21.15.73:46712) 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 -- 172.21.15.73:0/3204365573 learned_addr learned my addr 172.21.15.73:0/3204365573 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 -- 172.21.15.73:0/3204365573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 msgr2=0x55d79b15af50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 -- 172.21.15.73:0/3204365573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b15f6a0 msgr2=0x55d79b15b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b15f6a0 0x55d79b15b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 -- 172.21.15.73:0/3204365573 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d79b158270 con 0x55d79b1561a0 2024-03-09T00:13:38.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22bdddf640 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f22ac02c170 tx=0x7f22ac004ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f77176dc96bdaca server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.679+0000 7f22a37fe640 1 -- 172.21.15.73:0/3204365573 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac042040 con 0x55d79b1561a0 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22a37fe640 1 -- 172.21.15.73:0/3204365573 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22ac0046c0 con 0x55d79b1561a0 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22a37fe640 1 -- 172.21.15.73:0/3204365573 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac04d8b0 con 0x55d79b1561a0 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- 172.21.15.73:0/3204365573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 msgr2=0x55d79b15edb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 secure :-1 s=READY pgs=863 cs=0 l=1 rev1=1 crypto rx=0x7f22ac02c170 tx=0x7f22ac004ac0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- 172.21.15.73:0/3204365573 shutdown_connections 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b15f6a0 0x55d79b15b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b156ac0 0x55d79b15af50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.687 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/3204365573 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b15edb0 unknown :-1 s=CLOSED pgs=863 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- 172.21.15.73:0/3204365573 >> 172.21.15.73:0/3204365573 conn(0x55d79b1508d0 msgr2=0x55d79b176650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- 172.21.15.73:0/3204365573 shutdown_connections 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- 172.21.15.73:0/3204365573 wait complete. 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.680+0000 7f22c2c1c740 1 -- start start 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 Processor -- start 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b165630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b165b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.688 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81ad490 con 0x55d79b1561a0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81ae7b0 con 0x55d79b156ac0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22b81adaf0 con 0x55d79b15f6a0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22c2c1c740 10 client.0 _ll_register_callbacks cb 0x7f22ac005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46720/0 (socket says 172.21.15.73:46720) 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 -- 172.21.15.73:0/370429530 learned_addr learned my addr 172.21.15.73:0/370429530 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bd5de640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b165630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 -- 172.21.15.73:0/370429530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 msgr2=0x55d79b165b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b165b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 -- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 msgr2=0x55d79b165630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b165630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 -- 172.21.15.73:0/370429530 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d79b1c5de0 con 0x55d79b1561a0 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22bdddf640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f22ac04d530 tx=0x7f22ac02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9e5db7adbc42e6e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.681+0000 7f22a17fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f22ac051070 con 0x55d79b1561a0 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f22ac004030 con 0x55d79b1561a0 2024-03-09T00:13:38.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f22ac0046c0 con 0x55d79b1561a0 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f22ac042070 con 0x55d79b1561a0 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 10 client.16767 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d79b1c70e0 con 0x55d79b1561a0 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f22ac0046c0 con 0x55d79b1561a0 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f22ac04d630 con 0x55d79b1561a0 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 10 client.16767 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 10 client.16767 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:38.691 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 20 client.16767 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b1c72b0 0x55d79b1c96a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 4 client.16767 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55d79b0c1750 con 0x55d79b1c72b0 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.682+0000 7f22bd5de640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b1c72b0 0x55d79b1c96a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22bd5de640 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b1c72b0 0x55d79b1c96a0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=44f174c37c8e60a server_cookie=40376dfbed4ec5aa in_seq=0 out_seq=0 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22a17fa640 10 client.16767 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22a17fa640 1 -- 172.21.15.73:0/370429530 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55d79b0c1750 con 0x55d79b1c72b0 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22a17fa640 10 client.16767 handle_command_reply: tid=0 2024-03-09T00:13:38.692 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 client.16767 shutdown 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 msgr2=0x55d79b1650f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 crypto rx=0x7f22ac04d530 tx=0x7f22ac02bc30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 shutdown_connections 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d79b1c72b0 0x55d79b1c96a0 crc :-1 s=READY pgs=156 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d79b15f6a0 0x55d79b165b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d79b156ac0 0x55d79b165630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 --2- 172.21.15.73:0/370429530 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d79b1561a0 0x55d79b1650f0 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22a17fa640 0 client.16767 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:38.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 >> 172.21.15.73:0/370429530 conn(0x55d79b1508d0 msgr2=0x55d79b15ade0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 shutdown_connections 2024-03-09T00:13:38.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 1 -- 172.21.15.73:0/370429530 wait complete. 2024-03-09T00:13:38.694 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.683+0000 7f22c2c1c740 20 client.16767 trim_cache size 0 max 16384 2024-03-09T00:13:38.694 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 msgr2=0x7f22a40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 0x7f22a40a4e30 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f22b000af30 tx=0x7f22b000e7e0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 msgr2=0x7f22b819dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 secure :-1 s=READY pgs=860 cs=0 l=1 rev1=1 crypto rx=0x7f22b400c970 tx=0x7f22b400ce30 comp rx=0 tx=0).stop 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 shutdown_connections 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f22b81979e0 0x7f22b8197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f22a40a2970 0x7f22a40a4e30 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.696 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f22b81054b0 0x7f22b819dd20 unknown :-1 s=CLOSED pgs=860 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 --2- 172.21.15.73:0/2966172091 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f22b81003e0 0x7f22b819d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:38.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 >> 172.21.15.73:0/2966172091 conn(0x7f22b80773a0 msgr2=0x7f22b80fe250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:38.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 shutdown_connections 2024-03-09T00:13:38.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:38.684+0000 7f22c0043640 1 -- 172.21.15.73:0/2966172091 wait complete. 2024-03-09T00:13:39.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:39 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1357420176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:39.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:39 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1357420176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:39.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:39 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1357420176' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:13:39.698 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:13:39.699 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:13:39.699 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:13:39.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- 172.21.15.73:0/2239299770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac100410 msgr2=0x7f17ac1007f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/2239299770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac100410 0x7f17ac1007f0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 crypto rx=0x7f17a0009900 tx=0x7f17a002bc30 comp rx=0 tx=0).stop 2024-03-09T00:13:39.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- 172.21.15.73:0/2239299770 shutdown_connections 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/2239299770 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1054e0 0x7f17ac10d0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/2239299770 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100dc0 0x7f17ac104e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/2239299770 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac100410 0x7f17ac1007f0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- 172.21.15.73:0/2239299770 >> 172.21.15.73:0/2239299770 conn(0x7f17ac076570 msgr2=0x7f17ac076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- 172.21.15.73:0/2239299770 shutdown_connections 2024-03-09T00:13:39.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- 172.21.15.73:0/2239299770 wait complete. 2024-03-09T00:13:39.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 Processor -- start 2024-03-09T00:13:39.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.818+0000 7f17b2ce8640 1 -- start start 2024-03-09T00:13:39.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100410 0x7f17ac19d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1979f0 0x7f17ac197ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1116b0 con 0x7f17ac1054e0 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac111530 con 0x7f17ac100410 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac111830 con 0x7f17ac1979f0 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46732/0 (socket says 172.21.15.73:46732) 2024-03-09T00:13:39.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 -- 172.21.15.73:0/1435553885 learned_addr learned my addr 172.21.15.73:0/1435553885 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b0a84640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100410 0x7f17ac19d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b1285640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1979f0 0x7f17ac197ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 -- 172.21.15.73:0/1435553885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1979f0 msgr2=0x7f17ac197ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1979f0 0x7f17ac197ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 -- 172.21.15.73:0/1435553885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100410 msgr2=0x7f17ac19d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100410 0x7f17ac19d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.830 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 -- 172.21.15.73:0/1435553885 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac198510 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17abfff640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f179800c970 tx=0x7f179800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17a9ffb640 1 -- 172.21.15.73:0/1435553885 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1798007bb0 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17a9ffb640 1 -- 172.21.15.73:0/1435553885 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1798007d50 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac198800 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.819+0000 7f17a9ffb640 1 -- 172.21.15.73:0/1435553885 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179800f450 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.820+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac1a4a50 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17a9ffb640 1 -- 172.21.15.73:0/1435553885 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f17980046c0 con 0x7f17ac1054e0 2024-03-09T00:13:39.831 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 Processor -- start 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 -- start start 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17a9ffb640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 0x7f179c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498eb6d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb9670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e11b40 con 0x558498eadd80 2024-03-09T00:13:39.832 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498dbad80 con 0x558498eb7280 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b599e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498e0dbe0 con 0x558498eae6a0 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b1285640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 0x7f179c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b0a84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b0a84640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46738/0 (socket says 172.21.15.73:46738) 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17b0a84640 1 -- 172.21.15.73:0/1354082357 learned_addr learned my addr 172.21.15.73:0/1354082357 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.822+0000 7f17a9ffb640 1 -- 172.21.15.73:0/1435553885 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f17980cf330 con 0x7f17ac1054e0 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17abfff640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498eb6d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.833 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b1285640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb9670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 -- 172.21.15.73:0/1354082357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 msgr2=0x558498eb6d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498eb6d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 -- 172.21.15.73:0/1354082357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 msgr2=0x558498eb9670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb9670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 -- 172.21.15.73:0/1354082357 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558498bee7e0 con 0x558498eadd80 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b1285640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b1285640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 0x7f179c0a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f17a4005cc0 tx=0x7f17a400a710 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.834 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f17b0a84640 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f17a002cfd0 tx=0x7f17a002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=29fb9df0648e6762 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f1796ffd640 1 -- 172.21.15.73:0/1354082357 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0042070 con 0x558498eadd80 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f1796ffd640 1 -- 172.21.15.73:0/1354082357 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17a002cb00 con 0x558498eadd80 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.823+0000 7f1796ffd640 1 -- 172.21.15.73:0/1354082357 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a0005170 con 0x558498eadd80 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- 172.21.15.73:0/1354082357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 msgr2=0x558498eae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 crypto rx=0x7f17a002cfd0 tx=0x7f17a002c650 comp rx=0 tx=0).stop 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- 172.21.15.73:0/1354082357 shutdown_connections 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb9670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498eb6d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 --2- 172.21.15.73:0/1354082357 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eadd80 0x558498eae160 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- 172.21.15.73:0/1354082357 >> 172.21.15.73:0/1354082357 conn(0x558498ea84b0 msgr2=0x558498ea8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- 172.21.15.73:0/1354082357 shutdown_connections 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- 172.21.15.73:0/1354082357 wait complete. 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 -- start start 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.824+0000 7f17b599e740 1 Processor -- start 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb4790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498eba010 con 0x558498eae6a0 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498dce1c0 con 0x558498eb7280 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558498eba190 con 0x558498eadd80 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 10 client.0 _ll_register_callbacks cb 0x558498dbd7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46748/0 (socket says 172.21.15.73:46748) 2024-03-09T00:13:39.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 -- 172.21.15.73:0/4045837045 learned_addr learned my addr 172.21.15.73:0/4045837045 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b0a84640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b1285640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb4790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 -- 172.21.15.73:0/4045837045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 msgr2=0x558498eb6cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 -- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 msgr2=0x558498eb4790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb4790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 -- 172.21.15.73:0/4045837045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558498c4ef70 con 0x558498eae6a0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b1285640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17abfff640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1798007590 tx=0x7f179809afc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1db030e1bdf4f9a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f178bfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179801d050 con 0x558498eae6a0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f17980cf0b0 con 0x558498eae6a0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17a002cb00 con 0x558498eae6a0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f179809f6f0 con 0x558498eae6a0 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.825+0000 7f17b599e740 10 client.16779 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:39.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558498ae6170 con 0x558498eae6a0 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17a002cb00 con 0x558498eae6a0 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f17980b4250 con 0x558498eae6a0 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 10 client.16779 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 10 client.16779 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 20 client.16779 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498ec7250 0x558498ec9640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 4 client.16779 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:39.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558498de7960 con 0x558498ec7250 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.826+0000 7f17b0a84640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498ec7250 0x558498ec9640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b0a84640 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498ec7250 0x558498ec9640 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fbc298905a55dfa6 server_cookie=4d9ec6542f63a67f in_seq=0 out_seq=0 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f178bfff640 10 client.16779 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f178bfff640 1 -- 172.21.15.73:0/4045837045 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558498de7960 con 0x558498ec7250 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f178bfff640 10 client.16779 handle_command_reply: tid=0 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 client.16779 shutdown 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 msgr2=0x558498eb4050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 crypto rx=0x7f1798007590 tx=0x7f179809afc0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 shutdown_connections 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498ec7250 0x558498ec9640 crc :-1 s=READY pgs=157 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb4790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb4050 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f178bfff640 0 client.16779 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 --2- 172.21.15.73:0/4045837045 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.827+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 >> 172.21.15.73:0/4045837045 conn(0x558498ea84b0 msgr2=0x558498eaa200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.828+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 shutdown_connections 2024-03-09T00:13:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.828+0000 7f17b599e740 1 -- 172.21.15.73:0/4045837045 wait complete. 2024-03-09T00:13:39.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.828+0000 7f17b599e740 20 client.16779 trim_cache size 0 max 16384 2024-03-09T00:13:39.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 Processor -- start 2024-03-09T00:13:39.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 -- start start 2024-03-09T00:13:39.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb12d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1ac790 con 0x558498eae6a0 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1aed30 con 0x558498eb7280 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17b599e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1acd90 con 0x558498eadd80 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46750/0 (socket says 172.21.15.73:46750) 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 -- 172.21.15.73:0/1913372648 learned_addr learned my addr 172.21.15.73:0/1913372648 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.850 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 -- 172.21.15.73:0/1913372648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 msgr2=0x558498eb6780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 -- 172.21.15.73:0/1913372648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 msgr2=0x558498eb12d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb12d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 -- 172.21.15.73:0/1913372648 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558498eb4b00 con 0x558498eae6a0 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.843+0000 7f17abfff640 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1798097f70 tx=0x7f179809b720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dc73be6873e7f653 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17967fc640 1 -- 172.21.15.73:0/1913372648 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f179801d070 con 0x558498eae6a0 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17967fc640 1 -- 172.21.15.73:0/1913372648 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17980a3d40 con 0x558498eae6a0 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17967fc640 1 -- 172.21.15.73:0/1913372648 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17980cf040 con 0x558498eae6a0 2024-03-09T00:13:39.851 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- 172.21.15.73:0/1913372648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 msgr2=0x558498eb6cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 secure :-1 s=READY pgs=869 cs=0 l=1 rev1=1 crypto rx=0x7f1798097f70 tx=0x7f179809b720 comp rx=0 tx=0).stop 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- 172.21.15.73:0/1913372648 shutdown_connections 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eb7280 0x558498eb12d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eae6a0 0x558498eb6cc0 unknown :-1 s=CLOSED pgs=869 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 --2- 172.21.15.73:0/1913372648 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eadd80 0x558498eb6780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- 172.21.15.73:0/1913372648 >> 172.21.15.73:0/1913372648 conn(0x558498ea84b0 msgr2=0x558498eaf380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- 172.21.15.73:0/1913372648 shutdown_connections 2024-03-09T00:13:39.852 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- 172.21.15.73:0/1913372648 wait complete. 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.844+0000 7f17b599e740 1 -- start start 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 Processor -- start 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eadd80 0x558498ebc3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498ebc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1ad3b0 con 0x558498eb7280 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1ae6d0 con 0x558498eadd80 2024-03-09T00:13:39.853 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1ada10 con 0x558498eae6a0 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b599e740 10 client.0 _ll_register_callbacks cb 0x7f17a0009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46756/0 (socket says 172.21.15.73:46756) 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 -- 172.21.15.73:0/3516659026 learned_addr learned my addr 172.21.15.73:0/3516659026 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b0a84640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eadd80 0x558498ebc3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17abfff640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498ebc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 -- 172.21.15.73:0/3516659026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 msgr2=0x558498ebc8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.854 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498ebc8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 -- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eadd80 msgr2=0x558498ebc3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eadd80 0x558498ebc3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17b1285640 1 -- 172.21.15.73:0/3516659026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558498eae4d0 con 0x558498eb7280 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.845+0000 7f17abfff640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498ebc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b1285640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f17a400a510 tx=0x7f17a4004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88d22e6c9a8548c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f17a4023070 con 0x558498eb7280 2024-03-09T00:13:39.855 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f17a401f450 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f17980a3d40 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f17a402f050 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 10 client.16785 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558498f1fac0 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f17980a3d40 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f17a4027680 con 0x558498eb7280 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 10 client.16785 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:39.856 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 10 client.16785 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 20 client.16785 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498f20220 0x558498f22640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 4 client.16785 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.846+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x558498dd3080 con 0x558498f20220 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.847+0000 7f17b0a84640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498f20220 0x558498f22640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.847+0000 7f178bfff640 10 client.16785 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.847+0000 7f17b0a84640 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498f20220 0x558498f22640 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ed55bf1d7f56a8 server_cookie=3af85d75a39f25e6 in_seq=0 out_seq=0 2024-03-09T00:13:39.857 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f178bfff640 1 -- 172.21.15.73:0/3516659026 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x558498dd3080 con 0x558498f20220 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f178bfff640 10 client.16785 handle_command_reply: tid=0 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 client.16785 shutdown 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 msgr2=0x558498ebce30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 crypto rx=0x7f17a400a510 tx=0x7f17a4004a40 comp rx=0 tx=0).stop 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 shutdown_connections 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558498f20220 0x558498f22640 crc :-1 s=READY pgs=158 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558498eb7280 0x558498ebce30 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558498eae6a0 0x558498ebc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 --2- 172.21.15.73:0/3516659026 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558498eadd80 0x558498ebc3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 >> 172.21.15.73:0/3516659026 conn(0x558498ea84b0 msgr2=0x558498ea98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f178bfff640 0 client.16785 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 shutdown_connections 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 1 -- 172.21.15.73:0/3516659026 wait complete. 2024-03-09T00:13:39.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.848+0000 7f17b599e740 20 client.16785 trim_cache size 0 max 16384 2024-03-09T00:13:39.860 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:39.860 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:13:39.860 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "98fd23a7-2237-4791-86fb-80a2efd98504", 2024-03-09T00:13:39.860 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:13:39.860 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 msgr2=0x7f179c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 0x7f179c0a4e30 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f17a4005cc0 tx=0x7f17a400a710 comp rx=0 tx=0).stop 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 msgr2=0x7f17ac19dd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 secure :-1 s=READY pgs=866 cs=0 l=1 rev1=1 crypto rx=0x7f179800c970 tx=0x7f179800ce30 comp rx=0 tx=0).stop 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 shutdown_connections 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f17ac1979f0 0x7f17ac197ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f179c0a2970 0x7f179c0a4e30 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f17ac1054e0 0x7f17ac19dd80 unknown :-1 s=CLOSED pgs=866 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 --2- 172.21.15.73:0/1435553885 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f17ac100410 0x7f17ac19d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 >> 172.21.15.73:0/1435553885 conn(0x7f17ac076570 msgr2=0x7f17ac0fe120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 shutdown_connections 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.849+0000 7f17b2ce8640 1 -- 172.21.15.73:0/1435553885 wait complete. 2024-03-09T00:13:39.863 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:13:39.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:13:39.974 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/218475378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981054a0 msgr2=0x7f0d9810d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/218475378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981054a0 0x7f0d9810d070 secure :-1 s=READY pgs=871 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009ca0 tx=0x7f0d9002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/218475378 shutdown_connections 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/218475378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981054a0 0x7f0d9810d070 unknown :-1 s=CLOSED pgs=871 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/218475378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d98100d80 0x7f0d98104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/218475378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981003d0 0x7f0d981007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/218475378 >> 172.21.15.73:0/218475378 conn(0x7f0d980772f0 msgr2=0x7f0d98077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/218475378 shutdown_connections 2024-03-09T00:13:39.975 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.975+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/218475378 wait complete. 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 Processor -- start 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 -- start start 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 0x7f0d9819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d981978f0 0x7f0d98197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d98111670 con 0x7f0d981003d0 2024-03-09T00:13:39.976 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981114f0 con 0x7f0d981054a0 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9cbe1640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981117f0 con 0x7f0d981978f0 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46784/0 (socket says 172.21.15.73:46784) 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 -- 172.21.15.73:0/1352412953 learned_addr learned my addr 172.21.15.73:0/1352412953 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d981978f0 0x7f0d98197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 0x7f0d9819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 -- 172.21.15.73:0/1352412953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d981978f0 msgr2=0x7f0d98197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.977 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d981978f0 0x7f0d98197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 -- 172.21.15.73:0/1352412953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 msgr2=0x7f0d9819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 0x7f0d9819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d9659c640 1 -- 172.21.15.73:0/1352412953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d98198410 con 0x7f0d981003d0 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.976+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 0x7f0d9819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800e9f0 tx=0x7f0d8800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d8f7fe640 1 -- 172.21.15.73:0/1352412953 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d8800cd80 con 0x7f0d981003d0 2024-03-09T00:13:39.978 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d98198700 con 0x7f0d981003d0 2024-03-09T00:13:39.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d8f7fe640 1 -- 172.21.15.73:0/1352412953 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d88010910 con 0x7f0d981003d0 2024-03-09T00:13:39.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d8f7fe640 1 -- 172.21.15.73:0/1352412953 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d88018990 con 0x7f0d981003d0 2024-03-09T00:13:39.979 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.977+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d981a4950 con 0x7f0d981003d0 2024-03-09T00:13:39.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.979+0000 7f0d8f7fe640 1 -- 172.21.15.73:0/1352412953 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0d88010ac0 con 0x7f0d981003d0 2024-03-09T00:13:39.985 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 Processor -- start 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 -- start start 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6d7b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b63ae00 con 0x55aa4b6d8080 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b5e3b10 con 0x55aa4b6e0c60 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b636ea0 con 0x55aa4b6d7760 2024-03-09T00:13:39.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d8f7fe640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 0x7f0d840a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 0x7f0d840a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d96d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6d7b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46786/0 (socket says 172.21.15.73:46786) 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d9659c640 1 -- 172.21.15.73:0/171376010 learned_addr learned my addr 172.21.15.73:0/171376010 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.980+0000 7f0d8f7fe640 1 -- 172.21.15.73:0/1352412953 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0d880cf660 con 0x7f0d981003d0 2024-03-09T00:13:39.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 -- 172.21.15.73:0/171376010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 msgr2=0x55aa4b6d7b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6d7b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 -- 172.21.15.73:0/171376010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 msgr2=0x55aa4b6e3050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e3050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 -- 172.21.15.73:0/171376010 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa4b4197d0 con 0x55aa4b6d8080 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6d7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d9659c640 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0d880097b0 tx=0x7f0d8806b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4671291744bdf480 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 0x7f0d840a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0d90005e30 tx=0x7f0d9002e040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d8cff9640 1 -- 172.21.15.73:0/171376010 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d880ca0a0 con 0x55aa4b6d8080 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d8cff9640 1 -- 172.21.15.73:0/171376010 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d8809b8d0 con 0x55aa4b6d8080 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.981+0000 7f0d8cff9640 1 -- 172.21.15.73:0/171376010 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d880860d0 con 0x55aa4b6d8080 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/171376010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 msgr2=0x55aa4b6e0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 secure :-1 s=READY pgs=873 cs=0 l=1 rev1=1 crypto rx=0x7f0d880097b0 tx=0x7f0d8806b460 comp rx=0 tx=0).stop 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/171376010 shutdown_connections 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e3050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0720 unknown :-1 s=CLOSED pgs=873 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/171376010 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6d7b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/171376010 >> 172.21.15.73:0/171376010 conn(0x55aa4b6d1e90 msgr2=0x55aa4b6d2270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/171376010 shutdown_connections 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/171376010 wait complete. 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 -- start start 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.982+0000 7f0d9f7b7740 1 Processor -- start 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6dd7b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6ddcf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b6e3990 con 0x55aa4b6d7760 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b5f70a0 con 0x55aa4b6d8080 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55aa4b6e3b10 con 0x55aa4b6e0c60 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46788/0 (socket says 172.21.15.73:46788) 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 10 client.0 _ll_register_callbacks cb 0x55aa4b5f6a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 -- 172.21.15.73:0/3691346810 learned_addr learned my addr 172.21.15.73:0/3691346810 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6dd7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6ddcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 -- 172.21.15.73:0/3691346810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6e0c60 msgr2=0x55aa4b6ddcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6ddcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 -- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 msgr2=0x55aa4b6dd7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6dd7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 -- 172.21.15.73:0/3691346810 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa4b34f3f0 con 0x55aa4b6d7760 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9659c640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cfd0 tx=0x7f0d88086300 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6769ec3718801f38 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:39.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d880ccdb0 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0d880a3df0 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.983+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d8809b8d0 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d880a4070 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 10 client.16797 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55aa4b30f0f0 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d8809b8d0 con 0x55aa4b6d7760 2024-03-09T00:13:39.993 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0d88018b30 con 0x55aa4b6d7760 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 10 client.16797 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 10 client.16797 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 20 client.16797 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b6f0530 0x55aa4b6f2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 4 client.16797 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55aa4b648340 con 0x55aa4b6f0530 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.984+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b6f0530 0x55aa4b6f2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:39.994 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d7dffb640 10 client.16797 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b6f0530 0x55aa4b6f2920 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a20b58a925a8cf42 server_cookie=b7f312aa8ed36ea9 in_seq=0 out_seq=0 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d7dffb640 1 -- 172.21.15.73:0/3691346810 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55aa4b648340 con 0x55aa4b6f0530 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d7dffb640 10 client.16797 handle_command_reply: tid=0 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d9f7b7740 1 client.16797 shutdown 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 msgr2=0x55aa4b6e0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.985+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800cfd0 tx=0x7f0d88086300 comp rx=0 tx=0).stop 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 shutdown_connections 2024-03-09T00:13:39.995 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b6f0530 0x55aa4b6f2920 crc :-1 s=READY pgs=159 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6ddcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6dd7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d7dffb640 0 client.16797 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/3691346810 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e0420 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 >> 172.21.15.73:0/3691346810 conn(0x55aa4b6d1e90 msgr2=0x55aa4b6d3be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 shutdown_connections 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/3691346810 wait complete. 2024-03-09T00:13:39.996 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:39.986+0000 7f0d9f7b7740 20 client.16797 trim_cache size 0 max 16384 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 Processor -- start 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 -- start start 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6dfee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6daa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.003 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981ac8c0 con 0x55aa4b6d8080 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981aee60 con 0x55aa4b6e0c60 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981acec0 con 0x55aa4b6d7760 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46792/0 (socket says 172.21.15.73:46792) 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 -- 172.21.15.73:0/2989296300 learned_addr learned my addr 172.21.15.73:0/2989296300 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d9659c640 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6dfee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6daa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.004 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 -- 172.21.15.73:0/2989296300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 msgr2=0x55aa4b6dfee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6dfee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 -- 172.21.15.73:0/2989296300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 msgr2=0x55aa4b6daa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6daa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 -- 172.21.15.73:0/2989296300 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55aa4b6d7eb0 con 0x55aa4b6d8080 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.004+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009c00 tx=0x7f0d80014840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e5e9c20793dfaaa3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.005+0000 7f0d7ffff640 1 -- 172.21.15.73:0/2989296300 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d800142d0 con 0x55aa4b6d8080 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.005+0000 7f0d7ffff640 1 -- 172.21.15.73:0/2989296300 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d8001d390 con 0x55aa4b6d8080 2024-03-09T00:13:40.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.005+0000 7f0d7ffff640 1 -- 172.21.15.73:0/2989296300 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d80003c40 con 0x55aa4b6d8080 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.005+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/2989296300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 msgr2=0x55aa4b6e0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.005+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 crypto rx=0x7f0d80009c00 tx=0x7f0d80014840 comp rx=0 tx=0).stop 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/2989296300 shutdown_connections 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6daa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e0420 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/2989296300 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6dfee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/2989296300 >> 172.21.15.73:0/2989296300 conn(0x55aa4b6d1e90 msgr2=0x55aa4b6dc0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/2989296300 shutdown_connections 2024-03-09T00:13:40.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/2989296300 wait complete. 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- start start 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 Processor -- start 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e6710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e6c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981ad4e0 con 0x55aa4b6e0c60 2024-03-09T00:13:40.011 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981ae800 con 0x55aa4b6d8080 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d981adb40 con 0x55aa4b6d7760 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9f7b7740 10 client.0 _ll_register_callbacks cb 0x7f0d80009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d96d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d96d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:46802/0 (socket says 172.21.15.73:46802) 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d9659c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e6710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.006+0000 7f0d96d9d640 1 -- 172.21.15.73:0/1210839066 learned_addr learned my addr 172.21.15.73:0/1210839066 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d8ffff640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e6c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 -- 172.21.15.73:0/1210839066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 msgr2=0x55aa4b6e6710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:40.012 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e6710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 -- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 msgr2=0x55aa4b6e6c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e6c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 -- 172.21.15.73:0/1210839066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55aa4b746670 con 0x55aa4b6e0c60 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d96d9d640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009ca0 tx=0x7f0d9002c0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e7194e8f4087a185 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d7dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:13:40.013 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d90054070 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0d90004450 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d8001d390 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d90039040 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d9f7b7740 10 client.16803 fetch_fsmap learned FSMap version 20 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55aa4b7479b0 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.007+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d8001d390 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0d90034b20 con 0x55aa4b6e0c60 2024-03-09T00:13:40.014 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 10 client.16803 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 10 client.16803 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 20 client.16803 populate_metadata read hostname 'smithi073' 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b748140 0x55aa4b74a560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 4 client.16803 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55aa4b642690 con 0x55aa4b748140 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b748140 0x55aa4b74a560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d7dffb640 10 client.16803 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:40.015 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.008+0000 7f0d9659c640 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b748140 0x55aa4b74a560 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b65ba3fe9b6b6b4f server_cookie=9353d7d4fdb0c53c in_seq=0 out_seq=0 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d7dffb640 1 -- 172.21.15.73:0/1210839066 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55aa4b642690 con 0x55aa4b748140 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d7dffb640 10 client.16803 handle_command_reply: tid=0 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 client.16803 shutdown 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 msgr2=0x55aa4b6e7190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 secure :-1 s=READY pgs=876 cs=0 l=1 rev1=1 crypto rx=0x7f0d90009ca0 tx=0x7f0d9002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 shutdown_connections 2024-03-09T00:13:40.016 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55aa4b748140 0x55aa4b74a560 crc :-1 s=READY pgs=160 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d7dffb640 0 client.16803 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55aa4b6e0c60 0x55aa4b6e7190 unknown :-1 s=CLOSED pgs=876 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55aa4b6d8080 0x55aa4b6e6c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 --2- 172.21.15.73:0/1210839066 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55aa4b6d7760 0x55aa4b6e6710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 >> 172.21.15.73:0/1210839066 conn(0x55aa4b6d1e90 msgr2=0x55aa4b6d85a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 shutdown_connections 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 1 -- 172.21.15.73:0/1210839066 wait complete. 2024-03-09T00:13:40.017 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9f7b7740 20 client.16803 trim_cache size 0 max 16384 2024-03-09T00:13:40.018 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:13:40.018 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2535 inodes in the stack)", 2024-03-09T00:13:40.018 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:13:40.018 INFO:teuthology.orchestra.run.smithi073.stdout: "98fd23a7-2237-4791-86fb-80a2efd98504": { 2024-03-09T00:13:40.018 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:13:40.019 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "98fd23a7-2237-4791-86fb-80a2efd98504", 2024-03-09T00:13:40.019 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:13:40.019 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:13:40.019 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:13:40.019 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 msgr2=0x7f0d840a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 0x7f0d840a4e30 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f0d90005e30 tx=0x7f0d9002e040 comp rx=0 tx=0).stop 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 msgr2=0x7f0d9819d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.009+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 secure :-1 s=READY pgs=872 cs=0 l=1 rev1=1 crypto rx=0x7f0d8800e9f0 tx=0x7f0d8800eeb0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 shutdown_connections 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0d981978f0 0x7f0d98197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.021 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0d981054a0 0x7f0d9819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0d840a2970 0x7f0d840a4e30 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 --2- 172.21.15.73:0/1352412953 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0d981003d0 0x7f0d9819d740 unknown :-1 s=CLOSED pgs=872 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:13:40.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 >> 172.21.15.73:0/1352412953 conn(0x7f0d980772f0 msgr2=0x7f0d980fe1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:13:40.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 shutdown_connections 2024-03-09T00:13:40.022 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:13:40.010+0000 7f0d9cbe1640 1 -- 172.21.15.73:0/1352412953 wait complete. 2024-03-09T00:13:40.022 INFO:tasks.cephfs.filesystem:scrub status for tag:98fd23a7-2237-4791-86fb-80a2efd98504 - {'path': '/', 'tag': '98fd23a7-2237-4791-86fb-80a2efd98504', 'options': 'recursive,force'} 2024-03-09T00:13:40.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:40 smithi073 ceph-mon[32309]: pgmap v209: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:40.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:40 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:13:40.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:40 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:40.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:40 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:13:40.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:40 smithi102 ceph-mon[36106]: pgmap v209: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:40.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:40 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:13:40.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:40 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:40.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:40 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:13:40.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:40 smithi179 ceph-mon[36637]: pgmap v209: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:40.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:40 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:13:40.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:40 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:40.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:40 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:13:41.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:41 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:41.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:41 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:41.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:41 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:13:42.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:42 smithi102 ceph-mon[36106]: pgmap v210: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:42.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:42 smithi179 ceph-mon[36637]: pgmap v210: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:42.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:42 smithi073 ceph-mon[32309]: pgmap v210: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:43.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:43 smithi102 ceph-mon[36106]: pgmap v211: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:43.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:43 smithi179 ceph-mon[36637]: pgmap v211: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:43.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:43 smithi073 ceph-mon[32309]: pgmap v211: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:44.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:44 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:13:44.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:44 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:13:44.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:44 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:13:45.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:45 smithi102 ceph-mon[36106]: pgmap v212: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:45.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:45 smithi179 ceph-mon[36637]: pgmap v212: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:45.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:45 smithi073 ceph-mon[32309]: pgmap v212: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:48.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:48 smithi073 ceph-mon[32309]: pgmap v213: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:48.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:48 smithi102 ceph-mon[36106]: pgmap v213: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:48.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:48 smithi179 ceph-mon[36637]: pgmap v213: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:50.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:50 smithi073 ceph-mon[32309]: pgmap v214: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:50.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:50 smithi102 ceph-mon[36106]: pgmap v214: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:50.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:50 smithi179 ceph-mon[36637]: pgmap v214: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:52.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:52 smithi073 ceph-mon[32309]: pgmap v215: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:52.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:52 smithi102 ceph-mon[36106]: pgmap v215: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:52.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:52 smithi179 ceph-mon[36637]: pgmap v215: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:13:54.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:54 smithi073 ceph-mon[32309]: pgmap v216: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:54.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:54 smithi102 ceph-mon[36106]: pgmap v216: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:54.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:54 smithi179 ceph-mon[36637]: pgmap v216: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:13:56.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:56 smithi073 ceph-mon[32309]: pgmap v217: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:56.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:56 smithi102 ceph-mon[36106]: pgmap v217: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:56.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:56 smithi179 ceph-mon[36637]: pgmap v217: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:13:58.403 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:13:58 smithi073 ceph-mon[32309]: pgmap v218: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:58.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:13:58 smithi102 ceph-mon[36106]: pgmap v218: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:13:58.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:13:58 smithi179 ceph-mon[36637]: pgmap v218: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:00.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:00 smithi102 ceph-mon[36106]: pgmap v219: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:00.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:00 smithi179 ceph-mon[36637]: pgmap v219: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:00.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:00 smithi073 ceph-mon[32309]: pgmap v219: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:02.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:02 smithi102 ceph-mon[36106]: pgmap v220: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:14:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:14:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:14:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:14:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:02 smithi179 ceph-mon[36637]: pgmap v220: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:14:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:14:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:14:02.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:14:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:02 smithi073 ceph-mon[32309]: pgmap v220: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:14:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:14:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:14:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:14:04.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:04 smithi102 ceph-mon[36106]: pgmap v221: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:04.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:04 smithi179 ceph-mon[36637]: pgmap v221: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:04.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:04 smithi073 ceph-mon[32309]: pgmap v221: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:06.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:06 smithi102 ceph-mon[36106]: pgmap v222: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:14:06.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:06 smithi179 ceph-mon[36637]: pgmap v222: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:14:06.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:06 smithi073 ceph-mon[32309]: pgmap v222: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:14:07.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:07 smithi102 ceph-mon[36106]: pgmap v223: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:07.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:07 smithi179 ceph-mon[36637]: pgmap v223: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:07.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:07 smithi073 ceph-mon[32309]: pgmap v223: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:14:10.023 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:14:10.024 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:14:10.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.212+0000 7f8b95a8c640 1 -- 172.21.15.73:0/4153998229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b90100d80 msgr2=0x7f8b90104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.212+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/4153998229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b90100d80 0x7f8b90104dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f8b80009900 tx=0x7f8b8002bc80 comp rx=0 tx=0).stop 2024-03-09T00:14:10.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.214+0000 7f8b95a8c640 1 -- 172.21.15.73:0/4153998229 shutdown_connections 2024-03-09T00:14:10.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.214+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/4153998229 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b901054a0 0x7f8b9010d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.214+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/4153998229 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b90100d80 0x7f8b90104dc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.214+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/4153998229 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 0x7f8b901007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.214+0000 7f8b95a8c640 1 -- 172.21.15.73:0/4153998229 >> 172.21.15.73:0/4153998229 conn(0x7f8b900772f0 msgr2=0x7f8b90077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.217+0000 7f8b95a8c640 1 -- 172.21.15.73:0/4153998229 shutdown_connections 2024-03-09T00:14:10.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.218+0000 7f8b95a8c640 1 -- 172.21.15.73:0/4153998229 wait complete. 2024-03-09T00:14:10.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.219+0000 7f8b95a8c640 1 Processor -- start 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 -- start start 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 0x7f8b9019c800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b901054a0 0x7f8b9019cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90110680 con 0x7f8b90196a00 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90110500 con 0x7f8b901054a0 2024-03-09T00:14:10.219 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.221+0000 7f8b95a8c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90110800 con 0x7f8b901003d0 2024-03-09T00:14:10.220 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42284/0 (socket says 172.21.15.73:42284) 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 -- 172.21.15.73:0/1904952403 learned_addr learned my addr 172.21.15.73:0/1904952403 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 0x7f8b9019c800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 -- 172.21.15.73:0/1904952403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 msgr2=0x7f8b9019c800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 0x7f8b9019c800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 -- 172.21.15.73:0/1904952403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b901054a0 msgr2=0x7f8b9019cd40 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-09T00:14:10.221 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b901054a0 0x7f8b9019cd40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.222+0000 7f8b8ffff640 1 -- 172.21.15.73:0/1904952403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b90197520 con 0x7f8b90196a00 2024-03-09T00:14:10.222 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.223+0000 7f8b8ffff640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800db90 tx=0x7f8b8800dc80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.224+0000 7f8b8cff9640 1 -- 172.21.15.73:0/1904952403 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8800e980 con 0x7f8b90196a00 2024-03-09T00:14:10.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.224+0000 7f8b8cff9640 1 -- 172.21.15.73:0/1904952403 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b88009580 con 0x7f8b90196a00 2024-03-09T00:14:10.223 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.224+0000 7f8b8cff9640 1 -- 172.21.15.73:0/1904952403 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b88016650 con 0x7f8b90196a00 2024-03-09T00:14:10.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.225+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b901977b0 con 0x7f8b90196a00 2024-03-09T00:14:10.224 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.225+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b900686f0 con 0x7f8b90196a00 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.229+0000 7f8b98672740 1 Processor -- start 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 -- start start 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b8cff9640 1 -- 172.21.15.73:0/1904952403 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f8b8801d020 con 0x7f8b90196a00 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3a5c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d399940 0x55b21d399dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d3a1200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.228 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d302e00 con 0x55b21d3a5830 2024-03-09T00:14:10.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d2abb10 con 0x55b21d399940 2024-03-09T00:14:10.229 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.230+0000 7f8b98672740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d2feea0 con 0x55b21d39a300 2024-03-09T00:14:10.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8cff9640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 0x7f8b780a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d3a1200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3a5c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.230 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3a5c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42290/0 (socket says 172.21.15.73:42290) 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8effd640 1 -- 172.21.15.73:0/891366538 learned_addr learned my addr 172.21.15.73:0/891366538 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.231+0000 7f8b8cff9640 1 -- 172.21.15.73:0/1904952403 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f8b880ce510 con 0x7f8b90196a00 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.232+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d399940 0x55b21d399dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8effd640 1 -- 172.21.15.73:0/891366538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d399940 msgr2=0x55b21d399dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8effd640 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d399940 0x55b21d399dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8effd640 1 -- 172.21.15.73:0/891366538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 msgr2=0x55b21d3a5c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8effd640 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3a5c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8effd640 1 -- 172.21.15.73:0/891366538 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21d0e17d0 con 0x55b21d39a300 2024-03-09T00:14:10.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.233+0000 7f8b8ffff640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 0x7f8b780a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.234+0000 7f8b8ffff640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 0x7f8b780a5000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b880971b0 tx=0x7f8b88004b50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.237+0000 7f8b8effd640 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d3a1200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b80005ec0 tx=0x7f8b80000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=432f3ac116b90a4d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.238+0000 7f8b75ffb640 1 -- 172.21.15.73:0/891366538 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b8002ca80 con 0x55b21d39a300 2024-03-09T00:14:10.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.238+0000 7f8b75ffb640 1 -- 172.21.15.73:0/891366538 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b8002cc20 con 0x55b21d39a300 2024-03-09T00:14:10.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.239+0000 7f8b98672740 1 -- 172.21.15.73:0/891366538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 msgr2=0x55b21d3a1200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.239+0000 7f8b98672740 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d3a1200 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f8b80005ec0 tx=0x7f8b80000bc0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.242+0000 7f8b98672740 1 -- 172.21.15.73:0/891366538 shutdown_connections 2024-03-09T00:14:10.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.242+0000 7f8b98672740 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d3a1200 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.242+0000 7f8b98672740 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d399940 0x55b21d399dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.242+0000 7f8b98672740 1 --2- 172.21.15.73:0/891366538 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3a5c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.242+0000 7f8b98672740 1 -- 172.21.15.73:0/891366538 >> 172.21.15.73:0/891366538 conn(0x55b21d39c400 msgr2=0x55b21d39c7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.245+0000 7f8b98672740 1 -- 172.21.15.73:0/891366538 shutdown_connections 2024-03-09T00:14:10.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.246+0000 7f8b98672740 1 -- 172.21.15.73:0/891366538 wait complete. 2024-03-09T00:14:10.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.246+0000 7f8b98672740 1 -- start start 2024-03-09T00:14:10.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.246+0000 7f8b98672740 1 Processor -- start 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 0x55b21d3bf540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d39a300 0x55b21d3bfa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3b9980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d3a19d0 con 0x55b21d3a5830 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d2bf0a0 con 0x55b21d39a300 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b21d3a1b50 con 0x55b21d399940 2024-03-09T00:14:10.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b98672740 10 client.0 _ll_register_callbacks cb 0x55b21d2bea70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:10.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3b9980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3b9980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42300/0 (socket says 172.21.15.73:42300) 2024-03-09T00:14:10.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b8effd640 1 -- 172.21.15.73:0/3496426983 learned_addr learned my addr 172.21.15.73:0/3496426983 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.247+0000 7f8b8effd640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 0x55b21d3bf540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 -- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d39a300 msgr2=0x55b21d3bfa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d39a300 0x55b21d3bfa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 -- 172.21.15.73:0/3496426983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 msgr2=0x55b21d3b9980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3b9980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 -- 172.21.15.73:0/3496426983 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21d0173f0 con 0x55b21d399940 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.250+0000 7f8b8effd640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 0x55b21d3bf540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002cfd0 tx=0x7f8b80005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77e28df916253831 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.251+0000 7f8b6effd640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-03-09T00:14:10.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.251+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80004030 con 0x55b21d399940 2024-03-09T00:14:10.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.251+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b8002cc20 con 0x55b21d399940 2024-03-09T00:14:10.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.251+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f8b800041d0 con 0x55b21d399940 2024-03-09T00:14:10.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.253+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b800043d0 con 0x55b21d399940 2024-03-09T00:14:10.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.254+0000 7f8b98672740 10 client.26068 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:10.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.254+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55b21cfd70f0 con 0x55b21d399940 2024-03-09T00:14:10.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.256+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b8002cc20 con 0x55b21d399940 2024-03-09T00:14:10.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.256+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8b8004d040 con 0x55b21d399940 2024-03-09T00:14:10.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 10 client.26068 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:10.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 10 client.26068 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:10.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 20 client.26068 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:10.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d3c95c0 0x55b21d3cb9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 4 client.26068 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:10.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.257+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b21d310340 con 0x55b21d3c95c0 2024-03-09T00:14:10.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.259+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d3c95c0 0x55b21d3cb9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.261+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d3c95c0 0x55b21d3cb9b0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d2a334c4673136da server_cookie=20cda36bb611e485 in_seq=0 out_seq=0 2024-03-09T00:14:10.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.262+0000 7f8b6effd640 10 client.26068 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:10.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.263+0000 7f8b6effd640 1 -- 172.21.15.73:0/3496426983 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55b21d310340 con 0x55b21d3c95c0 2024-03-09T00:14:10.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.263+0000 7f8b6effd640 10 client.26068 handle_command_reply: tid=0 2024-03-09T00:14:10.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.263+0000 7f8b98672740 1 client.26068 shutdown 2024-03-09T00:14:10.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:10 smithi102 ceph-mon[36106]: osdmap e97: 12 total, 12 up, 12 in 2024-03-09T00:14:10.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:10 smithi102 ceph-mon[36106]: pgmap v225: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.265+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 msgr2=0x55b21d3bf540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.265+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 0x55b21d3bf540 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002cfd0 tx=0x7f8b80005fd0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 shutdown_connections 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d3c95c0 0x55b21d3cb9b0 crc :-1 s=READY pgs=161 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d3b9980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d39a300 0x55b21d3bfa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 --2- 172.21.15.73:0/3496426983 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d399940 0x55b21d3bf540 secure :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002cfd0 tx=0x7f8b80005fd0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.266+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 >> 172.21.15.73:0/3496426983 conn(0x55b21d39c400 msgr2=0x55b21d39bb60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.267+0000 7f8b6effd640 0 client.26068 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:10.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.269+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 shutdown_connections 2024-03-09T00:14:10.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.270+0000 7f8b98672740 1 -- 172.21.15.73:0/3496426983 wait complete. 2024-03-09T00:14:10.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.271+0000 7f8b98672740 20 client.26068 trim_cache size 0 max 16384 2024-03-09T00:14:10.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 Processor -- start 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 -- start start 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d39a300 0x55b21d39f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d3bf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90071ee0 con 0x55b21d39a300 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901b5630 con 0x55b21d3a4430 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.303+0000 7f8b98672740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b900724e0 con 0x55b21d3a5830 2024-03-09T00:14:10.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.304+0000 7f8b8f7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.304+0000 7f8b8f7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47472/0 (socket says 172.21.15.73:47472) 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.304+0000 7f8b8f7fe640 1 -- 172.21.15.73:0/4034228386 learned_addr learned my addr 172.21.15.73:0/4034228386 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.304+0000 7f8b8effd640 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d39a300 0x55b21d39f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.304+0000 7f8b8effd640 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d3bf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 -- 172.21.15.73:0/4034228386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 msgr2=0x55b21d3bf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d3bf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 -- 172.21.15.73:0/4034228386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d39a300 msgr2=0x55b21d39f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d39a300 0x55b21d39f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 -- 172.21.15.73:0/4034228386 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b21d394330 con 0x55b21d3a5830 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.305+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c680 tx=0x7f8b8400c6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff4ccf9e1a75f754 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.306+0000 7f8b757fa640 1 -- 172.21.15.73:0/4034228386 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b84017290 con 0x55b21d3a5830 2024-03-09T00:14:10.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.306+0000 7f8b757fa640 1 -- 172.21.15.73:0/4034228386 <== mon.1 v2:172.21.15.179:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b8401b070 con 0x55b21d3a5830 2024-03-09T00:14:10.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.307+0000 7f8b98672740 1 -- 172.21.15.73:0/4034228386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 msgr2=0x55b21d39f580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.307+0000 7f8b98672740 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f8b8400c680 tx=0x7f8b8400c6b0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.308+0000 7f8b98672740 1 -- 172.21.15.73:0/4034228386 shutdown_connections 2024-03-09T00:14:10.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.308+0000 7f8b98672740 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d3bf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.308+0000 7f8b98672740 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d3a5830 0x55b21d39f580 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.308+0000 7f8b98672740 1 --2- 172.21.15.73:0/4034228386 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d39a300 0x55b21d39f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.308+0000 7f8b98672740 1 -- 172.21.15.73:0/4034228386 >> 172.21.15.73:0/4034228386 conn(0x55b21d39c400 msgr2=0x55b21d3a0e30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.311+0000 7f8b98672740 1 -- 172.21.15.73:0/4034228386 shutdown_connections 2024-03-09T00:14:10.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.311+0000 7f8b98672740 1 -- 172.21.15.73:0/4034228386 wait complete. 2024-03-09T00:14:10.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 -- start start 2024-03-09T00:14:10.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 Processor -- start 2024-03-09T00:14:10.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d414450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d418510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90072b00 con 0x55b21d3a5830 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b901b4fd0 con 0x55b21d3a4430 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b90073160 con 0x55b21d39a300 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.312+0000 7f8b98672740 10 client.0 _ll_register_callbacks cb 0x7f8b8002c8d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47478/0 (socket says 172.21.15.73:47478) 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 -- 172.21.15.73:0/215221527 learned_addr learned my addr 172.21.15.73:0/215221527 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d418510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d414450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 -- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 msgr2=0x55b21d414450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d414450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 -- 172.21.15.73:0/215221527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 msgr2=0x55b21d418510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d418510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.313+0000 7f8b8effd640 1 -- 172.21.15.73:0/215221527 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b21d418b80 con 0x55b21d39a300 2024-03-09T00:14:10.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.314+0000 7f8b8effd640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002c7c0 tx=0x7f8b80003f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7db6880c3e63824f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.314+0000 7f8b6effd640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-03-09T00:14:10.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.315+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8b80003800 con 0x55b21d39a300 2024-03-09T00:14:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.315+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f8b80005210 con 0x55b21d39a300 2024-03-09T00:14:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.315+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8b8401b070 con 0x55b21d39a300 2024-03-09T00:14:10.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.315+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8b80005410 con 0x55b21d39a300 2024-03-09T00:14:10.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.317+0000 7f8b98672740 10 client.26074 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:10.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.318+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55b21d421130 con 0x55b21d39a300 2024-03-09T00:14:10.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.324+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8b8401b070 con 0x55b21d39a300 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8b80064030 con 0x55b21d39a300 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 10 client.26074 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 10 client.26074 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 20 client.26074 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d421740 0x55b21d423b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 4 client.26074 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.325+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55b21d30a690 con 0x55b21d421740 2024-03-09T00:14:10.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.326+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d421740 0x55b21d423b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.328+0000 7f8b8f7fe640 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d421740 0x55b21d423b30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a292597ee21c4e03 server_cookie=937539d11c7b0c36 in_seq=0 out_seq=0 2024-03-09T00:14:10.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.329+0000 7f8b6effd640 10 client.26074 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:10.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.331+0000 7f8b6effd640 1 -- 172.21.15.73:0/215221527 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55b21d30a690 con 0x55b21d421740 2024-03-09T00:14:10.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.331+0000 7f8b6effd640 10 client.26074 handle_command_reply: tid=0 2024-03-09T00:14:10.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.332+0000 7f8b98672740 1 client.26074 shutdown 2024-03-09T00:14:10.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.335+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 msgr2=0x55b21d413f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.335+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f8b8002c7c0 tx=0x7f8b80003f90 comp rx=0 tx=0).stop 2024-03-09T00:14:10.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:10 smithi179 ceph-mon[36637]: osdmap e97: 12 total, 12 up, 12 in 2024-03-09T00:14:10.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:10 smithi179 ceph-mon[36637]: pgmap v225: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 shutdown_connections 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55b21d421740 0x55b21d423b30 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55b21d3a5830 0x55b21d418510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55b21d3a4430 0x55b21d414450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 --2- 172.21.15.73:0/215221527 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55b21d39a300 0x55b21d413f10 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 >> 172.21.15.73:0/215221527 conn(0x55b21d39c400 msgr2=0x55b21d39b1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.337+0000 7f8b6effd640 0 client.26074 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:10.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.340+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 shutdown_connections 2024-03-09T00:14:10.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.340+0000 7f8b98672740 1 -- 172.21.15.73:0/215221527 wait complete. 2024-03-09T00:14:10.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.341+0000 7f8b98672740 20 client.26074 trim_cache size 0 max 16384 2024-03-09T00:14:10.339 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:10.339 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:14:10.340 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:14:10.340 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:10.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 msgr2=0x7f8b780a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.341 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 0x7f8b780a5000 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f8b880971b0 tx=0x7f8b88004b50 comp rx=0 tx=0).stop 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 msgr2=0x7f8b90196eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 crypto rx=0x7f8b8800db90 tx=0x7f8b8800dc80 comp rx=0 tx=0).stop 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 shutdown_connections 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f8b780a2b40 0x7f8b780a5000 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8b90196a00 0x7f8b90196eb0 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8b901054a0 0x7f8b9019cd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.342 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 --2- 172.21.15.73:0/1904952403 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f8b901003d0 0x7f8b9019c800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.343+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 >> 172.21.15.73:0/1904952403 conn(0x7f8b900772f0 msgr2=0x7f8b900fe340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.344+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 shutdown_connections 2024-03-09T00:14:10.343 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.344+0000 7f8b95a8c640 1 -- 172.21.15.73:0/1904952403 wait complete. 2024-03-09T00:14:10.362 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:14:10.362 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:14:10.363 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:10 smithi073 ceph-mon[32309]: osdmap e97: 12 total, 12 up, 12 in 2024-03-09T00:14:10.363 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:10 smithi073 ceph-mon[32309]: pgmap v225: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:10.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.544+0000 7f18dc880640 1 -- 172.21.15.73:0/2428249765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 msgr2=0x7f18d8100730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.542 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.544+0000 7f18dc880640 1 --2- 172.21.15.73:0/2428249765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8100730 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f18d0009ca0 tx=0x7f18d002c100 comp rx=0 tx=0).stop 2024-03-09T00:14:10.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.545+0000 7f18dc880640 1 -- 172.21.15.73:0/2428249765 shutdown_connections 2024-03-09T00:14:10.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.545+0000 7f18dc880640 1 --2- 172.21.15.73:0/2428249765 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f18d8105750 0x7f18d8105c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.545+0000 7f18dc880640 1 --2- 172.21.15.73:0/2428249765 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8100c70 0x7f18d8105130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.545+0000 7f18dc880640 1 --2- 172.21.15.73:0/2428249765 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8100730 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.543 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.545+0000 7f18dc880640 1 -- 172.21.15.73:0/2428249765 >> 172.21.15.73:0/2428249765 conn(0x7f18d80772f0 msgr2=0x7f18d8077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:10.544 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.546+0000 7f18dc880640 1 -- 172.21.15.73:0/2428249765 shutdown_connections 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.546+0000 7f18dc880640 1 -- 172.21.15.73:0/2428249765 wait complete. 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 Processor -- start 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 -- start start 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8072a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f18d8100c70 0x7f18d8072fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.546 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 0x7f18d806bd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8117dd0 con 0x7f18d8100c70 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8117c50 con 0x7f18d8100350 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18dc880640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18d8117f50 con 0x7f18d8105750 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18d6d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 0x7f18d806bd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18d6d9d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 0x7f18d806bd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47496/0 (socket says 172.21.15.73:47496) 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.547+0000 7f18d6d9d640 1 -- 172.21.15.73:0/3691006602 learned_addr learned my addr 172.21.15.73:0/3691006602 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:10.547 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.548+0000 7f18d659c640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8072a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.549 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 -- 172.21.15.73:0/3691006602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 msgr2=0x7f18d806bd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 0x7f18d806bd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 -- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f18d8100c70 msgr2=0x7f18d8072fd0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f18d8100c70 0x7f18d8072fd0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 -- 172.21.15.73:0/3691006602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d806c600 con 0x7f18d8100350 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18d659c640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8072a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18d002c710 tx=0x7f18d0034670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0039050 con 0x7f18d8100350 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.549+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f18d003a040 con 0x7f18d8100350 2024-03-09T00:14:10.550 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.550+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18d0039050 con 0x7f18d8100350 2024-03-09T00:14:10.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.550+0000 7f18dc880640 1 -- 172.21.15.73:0/3691006602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18d806c890 con 0x7f18d8100350 2024-03-09T00:14:10.551 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.550+0000 7f18dc880640 1 -- 172.21.15.73:0/3691006602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18d8070790 con 0x7f18d8100350 2024-03-09T00:14:10.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.556+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f18d0047020 con 0x7f18d8100350 2024-03-09T00:14:10.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.557+0000 7f18c77fe640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 0x7f18c00a52f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:10.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.557+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f18d0045020 con 0x7f18d8100350 2024-03-09T00:14:10.556 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.558+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18d8068c90 con 0x7f18d8100350 2024-03-09T00:14:10.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.559+0000 7f18d5d9b640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 0x7f18c00a52f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:10.560 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.560+0000 7f18d5d9b640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 0x7f18c00a52f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f18d806d2d0 tx=0x7f18cc00b370 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:10.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:10.568+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f18d004e060 con 0x7f18d8100350 2024-03-09T00:14:11.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.002+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18d806d040 con 0x7f18d8100350 2024-03-09T00:14:11.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.003+0000 7f18c77fe640 1 -- 172.21.15.73:0/3691006602 <== mon.2 v2:172.21.15.102:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f18d00bd3e0 con 0x7f18d8100350 2024-03-09T00:14:11.001 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:14:11.001 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:14:11.002 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 msgr2=0x7f18c00a52f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 0x7f18c00a52f0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f18d806d2d0 tx=0x7f18cc00b370 comp rx=0 tx=0).stop 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 msgr2=0x7f18d8072a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8072a90 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18d002c710 tx=0x7f18d0034670 comp rx=0 tx=0).stop 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 shutdown_connections 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f18d8105750 0x7f18d806bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.006 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f18c00a2e30 0x7f18c00a52f0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f18d8100c70 0x7f18d8072fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 --2- 172.21.15.73:0/3691006602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f18d8100350 0x7f18d8072a90 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.007+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 >> 172.21.15.73:0/3691006602 conn(0x7f18d80772f0 msgr2=0x7f18d8103100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.008+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 shutdown_connections 2024-03-09T00:14:11.007 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.008+0000 7f18c57fa640 1 -- 172.21.15.73:0/3691006602 wait complete. 2024-03-09T00:14:11.023 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:14:11.024 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:14:11.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 -- 172.21.15.73:0/3999018925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501054c0 msgr2=0x7fc850069960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 --2- 172.21.15.73:0/3999018925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501054c0 0x7fc850069960 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fc848009ca0 tx=0x7fc84802c140 comp rx=0 tx=0).stop 2024-03-09T00:14:11.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 -- 172.21.15.73:0/3999018925 shutdown_connections 2024-03-09T00:14:11.190 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 --2- 172.21.15.73:0/3999018925 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501054c0 0x7fc850069960 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 --2- 172.21.15.73:0/3999018925 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc850100da0 0x7fc850104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 --2- 172.21.15.73:0/3999018925 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501003f0 0x7fc8501007d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 -- 172.21.15.73:0/3999018925 >> 172.21.15.73:0/3999018925 conn(0x7fc850076570 msgr2=0x7fc850076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 -- 172.21.15.73:0/3999018925 shutdown_connections 2024-03-09T00:14:11.191 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.192+0000 7fc858214640 1 -- 172.21.15.73:0/3999018925 wait complete. 2024-03-09T00:14:11.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.193+0000 7fc858214640 1 Processor -- start 2024-03-09T00:14:11.193 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.193+0000 7fc858214640 1 -- start start 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 0x7fc850112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8501054c0 0x7fc850110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8501138d0 con 0x7fc8501054c0 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc850113750 con 0x7fc8501105d0 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc858214640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc850113a50 con 0x7fc8501003f0 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.194 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:34018/0 (socket says 172.21.15.73:34018) 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 -- 172.21.15.73:0/475445481 learned_addr learned my addr 172.21.15.73:0/475445481 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc855fb0640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 0x7fc850112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 -- 172.21.15.73:0/475445481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 msgr2=0x7fc850112ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 0x7fc850112ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 -- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8501054c0 msgr2=0x7fc850110090 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8501054c0 0x7fc850110090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc8567b1640 1 -- 172.21.15.73:0/475445481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc85019ffe0 con 0x7fc8501105d0 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.194+0000 7fc855fb0640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 0x7fc850112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:11.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.195+0000 7fc8567b1640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc848037040 tx=0x7fc848035ae0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.196+0000 7fc846ffd640 1 -- 172.21.15.73:0/475445481 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848039040 con 0x7fc8501105d0 2024-03-09T00:14:11.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.196+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8501a0270 con 0x7fc8501105d0 2024-03-09T00:14:11.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.196+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8501a0760 con 0x7fc8501105d0 2024-03-09T00:14:11.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.196+0000 7fc846ffd640 1 -- 172.21.15.73:0/475445481 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc848044070 con 0x7fc8501105d0 2024-03-09T00:14:11.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.196+0000 7fc846ffd640 1 -- 172.21.15.73:0/475445481 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc848004380 con 0x7fc8501105d0 2024-03-09T00:14:11.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.199+0000 7fc846ffd640 1 -- 172.21.15.73:0/475445481 <== mon.2 v2:172.21.15.102:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc848004520 con 0x7fc8501105d0 2024-03-09T00:14:11.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.200+0000 7fc85adf0740 1 Processor -- start 2024-03-09T00:14:11.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.200+0000 7fc85adf0740 1 -- start start 2024-03-09T00:14:11.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d54160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc8557af640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc8557af640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:34026/0 (socket says 172.21.15.73:34026) 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc8557af640 1 -- 172.21.15.73:0/4027272296 learned_addr learned my addr 172.21.15.73:0/4027272296 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc846ffd640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 0x7fc83c0a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc855fb0640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d54160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.201+0000 7fc85adf0740 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d56a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.202+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0d12360 con 0x5634f0d4bac0 2024-03-09T00:14:11.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.202+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0caeec0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.202+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0c57d30 con 0x5634f0d546a0 2024-03-09T00:14:11.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.203+0000 7fc8567b1640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d56a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.203+0000 7fc846ffd640 1 -- 172.21.15.73:0/475445481 <== mon.2 v2:172.21.15.102:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fc8480f1030 con 0x7fc8501105d0 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.203+0000 7fc8557af640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 0x7fc83c0a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 -- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 msgr2=0x5634f0d56a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d56a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 -- 172.21.15.73:0/4027272296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 msgr2=0x5634f0d54160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d54160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.202 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 -- 172.21.15.73:0/4027272296 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f0c56bc0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 0x7fc83c0a5000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8400125e0 tx=0x7fc840018000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8567b1640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d56a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:11.203 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.204+0000 7fc8557af640 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc8400146e0 tx=0x7fc840014ba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2216d6598a753bf server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.205+0000 7fc83bfff640 1 -- 172.21.15.73:0/4027272296 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84000f050 con 0x5634f0d4b1a0 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.205+0000 7fc83bfff640 1 -- 172.21.15.73:0/4027272296 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc840023070 con 0x5634f0d4b1a0 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.205+0000 7fc83bfff640 1 -- 172.21.15.73:0/4027272296 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84001e650 con 0x5634f0d4b1a0 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.206+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 msgr2=0x5634f0d4b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.206+0000 7fc85adf0740 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fc8400146e0 tx=0x7fc840014ba0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.204 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.206+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 shutdown_connections 2024-03-09T00:14:11.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.206+0000 7fc85adf0740 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d56a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.207+0000 7fc85adf0740 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d54160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.207+0000 7fc85adf0740 1 --2- 172.21.15.73:0/4027272296 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4b1a0 0x5634f0d4b580 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.207+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 >> 172.21.15.73:0/4027272296 conn(0x5634f0d458d0 msgr2=0x5634f0d45cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.205 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.207+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 shutdown_connections 2024-03-09T00:14:11.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.208+0000 7fc85adf0740 1 -- 172.21.15.73:0/4027272296 wait complete. 2024-03-09T00:14:11.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.209+0000 7fc85adf0740 1 -- start start 2024-03-09T00:14:11.206 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.209+0000 7fc85adf0740 1 Processor -- start 2024-03-09T00:14:11.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.210+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.208 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.210+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4bac0 0x5634f0d51bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.210+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d52110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc85adf0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0c4d4a0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42354/0 (socket says 172.21.15.73:42354) 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 -- 172.21.15.73:0/1543927706 learned_addr learned my addr 172.21.15.73:0/1543927706 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0caaf60 con 0x5634f0d546a0 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634f0d57430 con 0x5634f0d4bac0 2024-03-09T00:14:11.209 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc85adf0740 10 client.0 _ll_register_callbacks cb 0x5634f0ca8d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc8557af640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4bac0 0x5634f0d51bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 -- 172.21.15.73:0/1543927706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4bac0 msgr2=0x5634f0d51bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4bac0 0x5634f0d51bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 -- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 msgr2=0x5634f0d52110 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d52110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.210 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634f09c33f0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.211 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc855fb0640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc84c011800 tx=0x7fc84c011cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e98679afd127bce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.211+0000 7fc8397fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c010ba0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fc84c010d40 con 0x5634f0d4b1a0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc840023070 con 0x5634f0d4b1a0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc84c004020 con 0x5634f0d4b1a0 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc85adf0740 10 client.16824 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:11.212 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.212+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5634f0a8d7d0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84c0041c0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc84c01a590 con 0x5634f0d4b1a0 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 10 client.16824 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 10 client.16824 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 20 client.16824 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0d66a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 4 client.16824 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc840023070 con 0x5634f0d64650 2024-03-09T00:14:11.213 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.213+0000 7fc8557af640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0d66a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.214+0000 7fc8397fa640 10 client.16824 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:11.214 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.214+0000 7fc8557af640 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0d66a40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c0dbceccba5fc2ed server_cookie=778bcf35997408d5 in_seq=0 out_seq=0 2024-03-09T00:14:11.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc8397fa640 1 -- 172.21.15.73:0/1543927706 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7fc840023070 con 0x5634f0d64650 2024-03-09T00:14:11.215 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc8397fa640 10 client.16824 handle_command_reply: tid=0 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 client.16824 shutdown 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 msgr2=0x5634f0d51690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 crypto rx=0x7fc84c011800 tx=0x7fc84c011cc0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 shutdown_connections 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0d66a40 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d52110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.216 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.215+0000 7fc8397fa640 0 client.16824 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4bac0 0x5634f0d51bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1543927706 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d51690 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 >> 172.21.15.73:0/1543927706 conn(0x5634f0d458d0 msgr2=0x5634f0d47620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 shutdown_connections 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 1 -- 172.21.15.73:0/1543927706 wait complete. 2024-03-09T00:14:11.217 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.216+0000 7fc85adf0740 20 client.16824 trim_cache size 0 max 16384 2024-03-09T00:14:11.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.238+0000 7fc85adf0740 1 Processor -- start 2024-03-09T00:14:11.236 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.238+0000 7fc85adf0740 1 -- start start 2024-03-09T00:14:11.237 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.238+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.238+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 0x5634f0d51f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d4e8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42360/0 (socket says 172.21.15.73:42360) 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 -- 172.21.15.73:0/915493745 learned_addr learned my addr 172.21.15.73:0/915493745 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83400e1b0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc834010750 con 0x5634f0d4bac0 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83400e7b0 con 0x5634f0d546a0 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc8567b1640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d4e8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc8557af640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 0x5634f0d51f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 -- 172.21.15.73:0/915493745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 msgr2=0x5634f0d4e8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d4e8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 -- 172.21.15.73:0/915493745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 msgr2=0x5634f0d51f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 0x5634f0d51f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc855fb0640 1 -- 172.21.15.73:0/915493745 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634f0d56a90 con 0x5634f0d4b1a0 2024-03-09T00:14:11.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.239+0000 7fc8557af640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 0x5634f0d51f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:11.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.240+0000 7fc855fb0640 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc84c01ae60 tx=0x7fc84c011790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3925552b9f60e0a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.240+0000 7fc83b7fe640 1 -- 172.21.15.73:0/915493745 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c033070 con 0x5634f0d4b1a0 2024-03-09T00:14:11.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.240+0000 7fc83b7fe640 1 -- 172.21.15.73:0/915493745 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84c0043d0 con 0x5634f0d4b1a0 2024-03-09T00:14:11.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.240+0000 7fc83b7fe640 1 -- 172.21.15.73:0/915493745 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc84c005030 con 0x5634f0d4b1a0 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 msgr2=0x5634f0d53dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 secure :-1 s=READY pgs=879 cs=0 l=1 rev1=1 crypto rx=0x7fc84c01ae60 tx=0x7fc84c011790 comp rx=0 tx=0).stop 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 shutdown_connections 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d546a0 0x5634f0d4e8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d4bac0 0x5634f0d51f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 --2- 172.21.15.73:0/915493745 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4b1a0 0x5634f0d53dc0 unknown :-1 s=CLOSED pgs=879 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.241+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 >> 172.21.15.73:0/915493745 conn(0x5634f0d458d0 msgr2=0x5634f0d4c980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.242+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 shutdown_connections 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.242+0000 7fc85adf0740 1 -- 172.21.15.73:0/915493745 wait complete. 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.243+0000 7fc85adf0740 1 -- start start 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.243+0000 7fc85adf0740 1 Processor -- start 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.243+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 0x5634f0d5a140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.243+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d5a680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.243+0000 7fc85adf0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d5abc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc85adf0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83400edd0 con 0x5634f0d4bac0 2024-03-09T00:14:11.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc85adf0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8340100f0 con 0x5634f0d546a0 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc85adf0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83400f430 con 0x5634f0d4b1a0 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 0x5634f0d5a140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc855fb0640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 0x5634f0d5a140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47552/0 (socket says 172.21.15.73:47552) 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc855fb0640 1 -- 172.21.15.73:0/1818068354 learned_addr learned my addr 172.21.15.73:0/1818068354 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8557af640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d5a680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc85adf0740 10 client.0 _ll_register_callbacks cb 0x7fc840009060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:11.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d5abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 -- 172.21.15.73:0/1818068354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 msgr2=0x5634f0d5a140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 0x5634f0d5a140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 -- 172.21.15.73:0/1818068354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 msgr2=0x5634f0d5a680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d5a680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.244+0000 7fc8567b1640 1 -- 172.21.15.73:0/1818068354 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5634f0dbab80 con 0x5634f0d546a0 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.245+0000 7fc8567b1640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d5abc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8480ee3f0 tx=0x7fc8480b9660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c9d8ceb84a7748b8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.245+0000 7fc8397fa640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.245+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8480bda00 con 0x5634f0d546a0 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.245+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc84c0043d0 con 0x5634f0d546a0 2024-03-09T00:14:11.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.245+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fc8480c1150 con 0x5634f0d546a0 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.246+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc84803a250 con 0x5634f0d546a0 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.246+0000 7fc85adf0740 10 client.26063 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.246+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5634f0dbbec0 con 0x5634f0d546a0 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc84c0043d0 con 0x5634f0d546a0 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fc84803c030 con 0x5634f0d546a0 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc85adf0740 10 client.26063 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc85adf0740 10 client.26063 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:11.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc85adf0740 20 client.26063 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.247+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0dbe620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.248+0000 7fc85adf0740 4 client.26063 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.248+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5634f0cbc400 con 0x5634f0d64650 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.248+0000 7fc855fb0640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0dbe620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.249+0000 7fc855fb0640 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0dbe620 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=67441733b18fc18e server_cookie=18ad3327c36b7e in_seq=0 out_seq=0 2024-03-09T00:14:11.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.249+0000 7fc8397fa640 10 client.26063 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:11.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc8397fa640 1 -- 172.21.15.73:0/1818068354 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5634f0cbc400 con 0x5634f0d64650 2024-03-09T00:14:11.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc8397fa640 10 client.26063 handle_command_reply: tid=0 2024-03-09T00:14:11.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 client.26063 shutdown 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 msgr2=0x5634f0d5abc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d5abc0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fc8480ee3f0 tx=0x7fc8480b9660 comp rx=0 tx=0).stop 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 shutdown_connections 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5634f0d64650 0x5634f0dbe620 crc :-1 s=READY pgs=164 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5634f0d546a0 0x5634f0d5abc0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5634f0d4bac0 0x5634f0d5a680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 --2- 172.21.15.73:0/1818068354 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5634f0d4b1a0 0x5634f0d5a140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 >> 172.21.15.73:0/1818068354 conn(0x5634f0d458d0 msgr2=0x5634f0d44100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.250+0000 7fc8397fa640 0 client.26063 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:11.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.251+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 shutdown_connections 2024-03-09T00:14:11.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.251+0000 7fc85adf0740 1 -- 172.21.15.73:0/1818068354 wait complete. 2024-03-09T00:14:11.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.251+0000 7fc85adf0740 20 client.26063 trim_cache size 0 max 16384 2024-03-09T00:14:11.252 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 msgr2=0x7fc83c0a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 0x7fc83c0a5000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fc8400125e0 tx=0x7fc840018000 comp rx=0 tx=0).stop 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 msgr2=0x7fc850110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7fc848037040 tx=0x7fc848035ae0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 shutdown_connections 2024-03-09T00:14:11.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc8501105d0 0x7fc850110a60 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc83c0a2b40 0x7fc83c0a5000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc8501054c0 0x7fc850110090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 --2- 172.21.15.73:0/475445481 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc8501003f0 0x7fc850112ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.252+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 >> 172.21.15.73:0/475445481 conn(0x7fc850076570 msgr2=0x7fc8500feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.253+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 shutdown_connections 2024-03-09T00:14:11.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:11.253+0000 7fc858214640 1 -- 172.21.15.73:0/475445481 wait complete. 2024-03-09T00:14:11.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:11 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3691006602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:11.267 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:11 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3691006602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:11.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:11 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3691006602' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:12.269 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:14:12.269 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:14:12.269 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:14:12.295 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:12 smithi073 ceph-mon[32309]: pgmap v226: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:12.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:12 smithi179 ceph-mon[36637]: pgmap v226: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:12.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.381+0000 7ff613aa4640 1 -- 172.21.15.73:0/2219199469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c1065d0 msgr2=0x7ff60c10e1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.381+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2219199469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c1065d0 0x7ff60c10e1a0 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 crypto rx=0x7ff604009ca0 tx=0x7ff60402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- 172.21.15.73:0/2219199469 shutdown_connections 2024-03-09T00:14:12.382 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2219199469 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c1065d0 0x7ff60c10e1a0 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2219199469 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c101eb0 0x7ff60c105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2219199469 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c101500 0x7ff60c1018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- 172.21.15.73:0/2219199469 >> 172.21.15.73:0/2219199469 conn(0x7ff60c0fcd80 msgr2=0x7ff60c0ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- 172.21.15.73:0/2219199469 shutdown_connections 2024-03-09T00:14:12.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- 172.21.15.73:0/2219199469 wait complete. 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 Processor -- start 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- start start 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 0x7ff60c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 0x7ff60c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c111690 con 0x7ff60c101500 2024-03-09T00:14:12.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c111510 con 0x7ff60c1065d0 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff613aa4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c111810 con 0x7ff60c1979d0 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff611840640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff611840640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42394/0 (socket says 172.21.15.73:42394) 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.382+0000 7ff611840640 1 -- 172.21.15.73:0/2599136273 learned_addr learned my addr 172.21.15.73:0/2599136273 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff61103f640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 0x7ff60c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff612041640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 0x7ff60c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 -- 172.21.15.73:0/2599136273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 msgr2=0x7ff60c197e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 0x7ff60c197e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 -- 172.21.15.73:0/2599136273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 msgr2=0x7ff60c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 0x7ff60c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 -- 172.21.15.73:0/2599136273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff60c1984f0 con 0x7ff60c101500 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff61103f640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 0x7ff60c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff612041640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 0x7ff60c197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff611840640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff60800d770 tx=0x7ff60800dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.386 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff602ffd640 1 -- 172.21.15.73:0/2599136273 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608017070 con 0x7ff60c101500 2024-03-09T00:14:12.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff60c1987e0 con 0x7ff60c101500 2024-03-09T00:14:12.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff602ffd640 1 -- 172.21.15.73:0/2599136273 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff608005500 con 0x7ff60c101500 2024-03-09T00:14:12.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff602ffd640 1 -- 172.21.15.73:0/2599136273 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff608004e00 con 0x7ff60c101500 2024-03-09T00:14:12.387 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.383+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff60c1a49e0 con 0x7ff60c101500 2024-03-09T00:14:12.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.385+0000 7ff602ffd640 1 -- 172.21.15.73:0/2599136273 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff608004fa0 con 0x7ff60c101500 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.385+0000 7ff616767740 1 Processor -- start 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 -- start start 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff602ffd640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 0x7ff5f80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9d160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca8670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26c00b40 con 0x563b26c9d6a0 2024-03-09T00:14:12.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26ba9d80 con 0x563b26ca6280 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff616767740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26bfcbe0 con 0x563b26c9cd80 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff602ffd640 1 -- 172.21.15.73:0/2599136273 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7ff6080ceda0 con 0x7ff60c101500 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff612041640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 0x7ff5f80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42410/0 (socket says 172.21.15.73:42410) 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 -- 172.21.15.73:0/1032151991 learned_addr learned my addr 172.21.15.73:0/1032151991 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff611840640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff612041640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca8670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 -- 172.21.15.73:0/1032151991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 msgr2=0x563b26c9d160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9d160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 -- 172.21.15.73:0/1032151991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 msgr2=0x563b26ca8670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca8670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff61103f640 1 -- 172.21.15.73:0/1032151991 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b269dd7e0 con 0x563b26c9d6a0 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.386+0000 7ff612041640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff612041640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 0x7ff5f80a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x563b26c98640 tx=0x7ff604004a40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff61103f640 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009bf0 tx=0x7ff5fc0117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=84995840cc7ed7b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff5f7fff640 1 -- 172.21.15.73:0/1032151991 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc012450 con 0x563b26c9d6a0 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff5f7fff640 1 -- 172.21.15.73:0/1032151991 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc012ad0 con 0x563b26c9d6a0 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff5f7fff640 1 -- 172.21.15.73:0/1032151991 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc00fbc0 con 0x563b26c9d6a0 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff616767740 1 -- 172.21.15.73:0/1032151991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 msgr2=0x563b26ca5d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.387+0000 7ff616767740 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 secure :-1 s=READY pgs=882 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009bf0 tx=0x7ff5fc0117b0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- 172.21.15.73:0/1032151991 shutdown_connections 2024-03-09T00:14:12.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca8670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5d40 unknown :-1 s=CLOSED pgs=882 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- 172.21.15.73:0/1032151991 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9d160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- 172.21.15.73:0/1032151991 >> 172.21.15.73:0/1032151991 conn(0x563b26c974b0 msgr2=0x563b26c97890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- 172.21.15.73:0/1032151991 shutdown_connections 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- 172.21.15.73:0/1032151991 wait complete. 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- start start 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 Processor -- start 2024-03-09T00:14:12.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26ca5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9d6a0 0x563b26ca3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26ca9070 con 0x563b26ca6280 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26bbd1c0 con 0x563b26c9d6a0 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563b26ca91f0 con 0x563b26c9cd80 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.388+0000 7ff616767740 10 client.0 _ll_register_callbacks cb 0x563b26bac7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42416/0 (socket says 172.21.15.73:42416) 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 -- 172.21.15.73:0/2438093009 learned_addr learned my addr 172.21.15.73:0/2438093009 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff611840640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26ca5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff61103f640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9d6a0 0x563b26ca3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 -- 172.21.15.73:0/2438093009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 msgr2=0x563b26ca5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26ca5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 -- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9d6a0 msgr2=0x563b26ca3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9d6a0 0x563b26ca3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 -- 172.21.15.73:0/2438093009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b26a3df70 con 0x563b26ca6280 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff612041640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff6040095c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1993661d51dcf8ea server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff5f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604039050 con 0x563b26ca6280 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7ff604034440 con 0x563b26ca6280 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5fc012ad0 con 0x563b26ca6280 2024-03-09T00:14:12.401 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60405a050 con 0x563b26ca6280 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff616767740 10 client.16842 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563b268d5170 con 0x563b26ca6280 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.389+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc012ad0 con 0x563b26ca6280 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff604055030 con 0x563b26ca6280 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 10 client.16842 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:12.402 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 10 client.16842 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 20 client.16842 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26cb6320 0x563b26cb8730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 4 client.16842 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563b26bd6960 con 0x563b26cb6320 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff611840640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26cb6320 0x563b26cb8730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff5f57fa640 10 client.16842 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.390+0000 7ff611840640 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26cb6320 0x563b26cb8730 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8e9fd81505fa8dd server_cookie=c92c8ff0748d0bdd in_seq=0 out_seq=0 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff5f57fa640 1 -- 172.21.15.73:0/2438093009 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563b26bd6960 con 0x563b26cb6320 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff5f57fa640 10 client.16842 handle_command_reply: tid=0 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 client.16842 shutdown 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 msgr2=0x563b26ca37a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff6040095c0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.404 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 shutdown_connections 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26cb6320 0x563b26cb8730 crc :-1 s=READY pgs=165 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26ca37a0 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9d6a0 0x563b26ca3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff5f57fa640 0 client.16842 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 --2- 172.21.15.73:0/2438093009 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26ca5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 >> 172.21.15.73:0/2438093009 conn(0x563b26c974b0 msgr2=0x563b26c99200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 shutdown_connections 2024-03-09T00:14:12.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 1 -- 172.21.15.73:0/2438093009 wait complete. 2024-03-09T00:14:12.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.391+0000 7ff616767740 20 client.16842 trim_cache size 0 max 16384 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 Processor -- start 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 -- start start 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca5cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1ac940 con 0x563b26c9d6a0 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1aeee0 con 0x563b26ca6280 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff616767740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1acf40 con 0x563b26c9cd80 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff61103f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff61103f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42420/0 (socket says 172.21.15.73:42420) 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.406+0000 7ff61103f640 1 -- 172.21.15.73:0/2256682822 learned_addr learned my addr 172.21.15.73:0/2256682822 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff612041640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca5cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff611840640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 -- 172.21.15.73:0/2256682822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 msgr2=0x563b26c9da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 -- 172.21.15.73:0/2256682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 msgr2=0x563b26ca5cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca5cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 -- 172.21.15.73:0/2256682822 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563b26c9d4d0 con 0x563b26c9d6a0 2024-03-09T00:14:12.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff612041640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff611840640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff61103f640 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009bf0 tx=0x7ff5fc0098d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=95f58928fe2c0637 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff5f77fe640 1 -- 172.21.15.73:0/2256682822 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc029030 con 0x563b26c9d6a0 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff5f77fe640 1 -- 172.21.15.73:0/2256682822 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc005520 con 0x563b26c9d6a0 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.407+0000 7ff5f77fe640 1 -- 172.21.15.73:0/2256682822 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5fc003cd0 con 0x563b26c9d6a0 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- 172.21.15.73:0/2256682822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 msgr2=0x563b26ca5790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 secure :-1 s=READY pgs=884 cs=0 l=1 rev1=1 crypto rx=0x7ff5fc009bf0 tx=0x7ff5fc0098d0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- 172.21.15.73:0/2256682822 shutdown_connections 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26ca6280 0x563b26ca5cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26c9d6a0 0x563b26ca5790 unknown :-1 s=CLOSED pgs=884 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- 172.21.15.73:0/2256682822 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9cd80 0x563b26c9da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- 172.21.15.73:0/2256682822 >> 172.21.15.73:0/2256682822 conn(0x563b26c974b0 msgr2=0x563b26c9e380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- 172.21.15.73:0/2256682822 shutdown_connections 2024-03-09T00:14:12.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- 172.21.15.73:0/2256682822 wait complete. 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- start start 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 Processor -- start 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9cd80 0x563b26cad760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9d6a0 0x563b26caaaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1ad560 con 0x563b26ca6280 2024-03-09T00:14:12.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1ae880 con 0x563b26c9cd80 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff60c1adbc0 con 0x563b26c9d6a0 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.408+0000 7ff616767740 10 client.0 _ll_register_callbacks cb 0x7ff5fc009590 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42436/0 (socket says 172.21.15.73:42436) 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 -- 172.21.15.73:0/543857968 learned_addr learned my addr 172.21.15.73:0/543857968 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff61103f640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9d6a0 0x563b26caaaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff611840640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9cd80 0x563b26cad760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 -- 172.21.15.73:0/543857968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9d6a0 msgr2=0x563b26caaaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9d6a0 0x563b26caaaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 -- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9cd80 msgr2=0x563b26cad760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9cd80 0x563b26cad760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 -- 172.21.15.73:0/543857968 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563b26cab7a0 con 0x563b26ca6280 2024-03-09T00:14:12.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff612041640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff60402ecc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d092c1376350729 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff604039050 con 0x563b26ca6280 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7ff604042650 con 0x563b26ca6280 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5fc005520 con 0x563b26ca6280 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff60406e050 con 0x563b26ca6280 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff616767740 10 client.16848 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:12.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563b26d0e770 con 0x563b26ca6280 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5fc005520 con 0x563b26ca6280 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff604055040 con 0x563b26ca6280 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.409+0000 7ff616767740 10 client.16848 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff616767740 10 client.16848 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff616767740 20 client.16848 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:12.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26d0ee30 0x563b26d11250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff616767740 4 client.16848 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x563b26bc2080 con 0x563b26d0ee30 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff611840640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26d0ee30 0x563b26d11250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff5f57fa640 10 client.16848 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.410+0000 7ff611840640 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26d0ee30 0x563b26d11250 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5747ba906f353f65 server_cookie=7ff927117d07185 in_seq=0 out_seq=0 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff5f57fa640 1 -- 172.21.15.73:0/543857968 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x563b26bc2080 con 0x563b26d0ee30 2024-03-09T00:14:12.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff5f57fa640 10 client.16848 handle_command_reply: tid=0 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 client.16848 shutdown 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 msgr2=0x563b26cab030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 crypto rx=0x7ff604036000 tx=0x7ff60402ecc0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 shutdown_connections 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563b26d0ee30 0x563b26d11250 crc :-1 s=READY pgs=166 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563b26ca6280 0x563b26cab030 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563b26c9d6a0 0x563b26caaaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff5f57fa640 0 client.16848 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 --2- 172.21.15.73:0/543857968 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563b26c9cd80 0x563b26cad760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 >> 172.21.15.73:0/543857968 conn(0x563b26c974b0 msgr2=0x563b26c95bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 shutdown_connections 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.411+0000 7ff616767740 1 -- 172.21.15.73:0/543857968 wait complete. 2024-03-09T00:14:12.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff616767740 20 client.16848 trim_cache size 0 max 16384 2024-03-09T00:14:12.426 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:12.426 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:14:12.426 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "90a86510-42c9-4cf1-9ecc-7ecd22a3bab9", 2024-03-09T00:14:12.426 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:14:12.426 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:12.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 msgr2=0x7ff5f80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 0x7ff5f80a4e30 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x563b26c98640 tx=0x7ff604004a40 comp rx=0 tx=0).stop 2024-03-09T00:14:12.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 msgr2=0x7ff60c19d820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 crypto rx=0x7ff60800d770 tx=0x7ff60800dc30 comp rx=0 tx=0).stop 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 shutdown_connections 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff60c1979d0 0x7ff60c197e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff60c1065d0 0x7ff60c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5f80a2970 0x7ff5f80a4e30 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 --2- 172.21.15.73:0/2599136273 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff60c101500 0x7ff60c19d820 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 >> 172.21.15.73:0/2599136273 conn(0x7ff60c0fcd80 msgr2=0x7ff60c0ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 shutdown_connections 2024-03-09T00:14:12.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.412+0000 7ff613aa4640 1 -- 172.21.15.73:0/2599136273 wait complete. 2024-03-09T00:14:12.430 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:14:12.430 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:14:12.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:12 smithi102 ceph-mon[36106]: pgmap v226: 129 pgs: 129 active+clean; 35 MiB data, 831 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:14:12.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 -- 172.21.15.73:0/374801970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101eb0 msgr2=0x7fce98105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 --2- 172.21.15.73:0/374801970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101eb0 0x7fce98105ef0 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 crypto rx=0x7fce8c009900 tx=0x7fce8c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 -- 172.21.15.73:0/374801970 shutdown_connections 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 --2- 172.21.15.73:0/374801970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 0x7fce9810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 --2- 172.21.15.73:0/374801970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101eb0 0x7fce98105ef0 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 --2- 172.21.15.73:0/374801970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98101500 0x7fce981018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 -- 172.21.15.73:0/374801970 >> 172.21.15.73:0/374801970 conn(0x7fce980fcd20 msgr2=0x7fce980ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 -- 172.21.15.73:0/374801970 shutdown_connections 2024-03-09T00:14:12.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.585+0000 7fce9e620640 1 -- 172.21.15.73:0/374801970 wait complete. 2024-03-09T00:14:12.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 Processor -- start 2024-03-09T00:14:12.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 -- start start 2024-03-09T00:14:12.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 0x7fce9819de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98197b00 0x7fce98197fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce98111660 con 0x7fce98101500 2024-03-09T00:14:12.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981114e0 con 0x7fce981065d0 2024-03-09T00:14:12.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.586+0000 7fce9e620640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981117e0 con 0x7fce98197b00 2024-03-09T00:14:12.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce977fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 0x7fce9819de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42446/0 (socket says 172.21.15.73:42446) 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 -- 172.21.15.73:0/1696453110 learned_addr learned my addr 172.21.15.73:0/1696453110 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98197b00 0x7fce98197fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 -- 172.21.15.73:0/1696453110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98197b00 msgr2=0x7fce98197fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98197b00 0x7fce98197fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 -- 172.21.15.73:0/1696453110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 msgr2=0x7fce9819de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 0x7fce9819de40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 -- 172.21.15.73:0/1696453110 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce98198620 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce97fff640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fce8800e9f0 tx=0x7fce8800edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce957fa640 1 -- 172.21.15.73:0/1696453110 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8800cd80 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce957fa640 1 -- 172.21.15.73:0/1696453110 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce88005500 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce98198910 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce957fa640 1 -- 172.21.15.73:0/1696453110 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce88010690 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.587+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce981a4ac0 con 0x7fce98101500 2024-03-09T00:14:12.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fce957fa640 1 -- 172.21.15.73:0/1696453110 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fce880046c0 con 0x7fce98101500 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 Processor -- start 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 -- start start 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93235050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a9318ce00 con 0x563a93229760 2024-03-09T00:14:12.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a93135b10 con 0x563a9322a080 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fcea1201740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a93188ea0 con 0x563a93232c60 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fce9cbbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fce9cbbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42454/0 (socket says 172.21.15.73:42454) 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.590+0000 7fce9cbbd640 1 -- 172.21.15.73:0/1102001211 learned_addr learned my addr 172.21.15.73:0/1102001211 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce97fff640 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce977fe640 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93235050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce957fa640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 0x7fce800a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 -- 172.21.15.73:0/1102001211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 msgr2=0x563a93235050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93235050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 -- 172.21.15.73:0/1102001211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 msgr2=0x563a93232720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 -- 172.21.15.73:0/1102001211 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a92f6b7d0 con 0x563a93229760 2024-03-09T00:14:12.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce957fa640 1 -- 172.21.15.73:0/1696453110 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fce880cf6a0 con 0x7fce98101500 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 0x7fce800a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fce900104a0 tx=0x7fce90010960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dd7530d17bdec4b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce7e7fc640 1 -- 172.21.15.73:0/1102001211 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9001d910 con 0x563a93229760 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce7e7fc640 1 -- 172.21.15.73:0/1102001211 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce90011440 con 0x563a93229760 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 0x7fce800a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce90021000 tx=0x7fce9001e000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.591+0000 7fce7e7fc640 1 -- 172.21.15.73:0/1102001211 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce9001c6a0 con 0x563a93229760 2024-03-09T00:14:12.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- 172.21.15.73:0/1102001211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 msgr2=0x563a93229b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 secure :-1 s=READY pgs=888 cs=0 l=1 rev1=1 crypto rx=0x7fce900104a0 tx=0x7fce90010960 comp rx=0 tx=0).stop 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- 172.21.15.73:0/1102001211 shutdown_connections 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93235050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 --2- 172.21.15.73:0/1102001211 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93229b40 unknown :-1 s=CLOSED pgs=888 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- 172.21.15.73:0/1102001211 >> 172.21.15.73:0/1102001211 conn(0x563a93223e90 msgr2=0x563a93224270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- 172.21.15.73:0/1102001211 shutdown_connections 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- 172.21.15.73:0/1102001211 wait complete. 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.592+0000 7fcea1201740 1 -- start start 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 Processor -- start 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a93235a50 con 0x563a9322a080 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a931490a0 con 0x563a93229760 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x563a93235bd0 con 0x563a93232c60 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fcea1201740 10 client.0 _ll_register_callbacks cb 0x563a93148a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce977fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce977fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42456/0 (socket says 172.21.15.73:42456) 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:34158/0 (socket says 172.21.15.73:34158) 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce977fe640 1 -- 172.21.15.73:0/2832413223 learned_addr learned my addr 172.21.15.73:0/2832413223 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce9cbbd640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 -- 172.21.15.73:0/2832413223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 msgr2=0x563a9322fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322fcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 -- 172.21.15.73:0/2832413223 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 msgr2=0x563a9322f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce97fff640 1 -- 172.21.15.73:0/2832413223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a92ea13f0 con 0x563a93229760 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.593+0000 7fce977fe640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce97fff640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce880cc9b0 tx=0x7fce88009310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=892ae3a218e40b12 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8809b220 con 0x563a93229760 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce90011440 con 0x563a93229760 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fce880b46f0 con 0x563a93229760 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce880b9050 con 0x563a93229760 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 10 client.26090 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:12.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563a92e610f0 con 0x563a93229760 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce90011440 con 0x563a93229760 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fce880860b0 con 0x563a93229760 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 10 client.26090 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 10 client.26090 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 20 client.26090 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a93244bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 4 client.26090 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:12.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.594+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563a9319a340 con 0x563a93242780 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.595+0000 7fce977fe640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a93244bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.595+0000 7fce977fe640 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a93244bb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d27bd580360f9f64 server_cookie=c3350c349297bd66 in_seq=0 out_seq=0 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.595+0000 7fce777fe640 10 client.26090 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fce777fe640 1 -- 172.21.15.73:0/2832413223 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x563a9319a340 con 0x563a93242780 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fce777fe640 10 client.26090 handle_command_reply: tid=0 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 client.26090 shutdown 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 msgr2=0x563a932323e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fce880cc9b0 tx=0x7fce88009310 comp rx=0 tx=0).stop 2024-03-09T00:14:12.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 shutdown_connections 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a93244bb0 crc :-1 s=READY pgs=167 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fce777fe640 0 client.26090 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a9322a080 0x563a9322f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 --2- 172.21.15.73:0/2832413223 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a93229760 0x563a932323e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 >> 172.21.15.73:0/2832413223 conn(0x563a93223e90 msgr2=0x563a93225be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 shutdown_connections 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 1 -- 172.21.15.73:0/2832413223 wait complete. 2024-03-09T00:14:12.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.596+0000 7fcea1201740 20 client.26090 trim_cache size 0 max 16384 2024-03-09T00:14:12.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.614+0000 7fcea1201740 1 Processor -- start 2024-03-09T00:14:12.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.614+0000 7fcea1201740 1 -- start start 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.614+0000 7fcea1201740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fcea1201740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fcea1201740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322ca10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fcea1201740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981ac940 con 0x563a93229760 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fcea1201740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981aeee0 con 0x563a9322a080 2024-03-09T00:14:12.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fcea1201740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981acf40 con 0x563a93232c60 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42460/0 (socket says 172.21.15.73:42460) 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 -- 172.21.15.73:0/855416345 learned_addr learned my addr 172.21.15.73:0/855416345 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce977fe640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322ca10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 -- 172.21.15.73:0/855416345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 msgr2=0x563a9322ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322ca10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 -- 172.21.15.73:0/855416345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 msgr2=0x563a93232370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 -- 172.21.15.73:0/855416345 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563a93235930 con 0x563a93229760 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce97fff640 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fce880ccfb0 tx=0x7fce88004bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bd468288cb30a5f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce7dffb640 1 -- 172.21.15.73:0/855416345 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880ac070 con 0x563a93229760 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce7dffb640 1 -- 172.21.15.73:0/855416345 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce880d5030 con 0x563a93229760 2024-03-09T00:14:12.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.615+0000 7fce7dffb640 1 -- 172.21.15.73:0/855416345 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce8809fa10 con 0x563a93229760 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- 172.21.15.73:0/855416345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 msgr2=0x563a93231e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 crypto rx=0x7fce880ccfb0 tx=0x7fce88004bc0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- 172.21.15.73:0/855416345 shutdown_connections 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a9322ca10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93232370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 --2- 172.21.15.73:0/855416345 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93231e30 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- 172.21.15.73:0/855416345 >> 172.21.15.73:0/855416345 conn(0x563a93223e90 msgr2=0x563a9322e060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- 172.21.15.73:0/855416345 shutdown_connections 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- 172.21.15.73:0/855416345 wait complete. 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.616+0000 7fcea1201740 1 -- start start 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 Processor -- start 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93297740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93291610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981ad560 con 0x563a93229760 2024-03-09T00:14:12.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981ae880 con 0x563a9322a080 2024-03-09T00:14:12.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce981adbc0 con 0x563a93232c60 2024-03-09T00:14:12.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fcea1201740 10 client.0 _ll_register_callbacks cb 0x7fce8809b780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:12.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42472/0 (socket says 172.21.15.73:42472) 2024-03-09T00:14:12.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 -- 172.21.15.73:0/995169817 learned_addr learned my addr 172.21.15.73:0/995169817 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:12.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce977fe640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93297740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce9cbbd640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93291610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 -- 172.21.15.73:0/995169817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 msgr2=0x563a93291610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93291610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 -- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 msgr2=0x563a93297740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:12.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93297740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.617+0000 7fce97fff640 1 -- 172.21.15.73:0/995169817 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563a93291d00 con 0x563a93229760 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce97fff640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fce8809b850 tx=0x7fce8809b8c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=267d8e7be0350cfa server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fce880ac050 con 0x563a93229760 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(97..97 src has 1..97) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fce8809af20 con 0x563a93229760 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fce880d5030 con 0x563a93229760 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fce880a3c70 con 0x563a93229760 2024-03-09T00:14:12.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 10 client.16866 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x563a93291d00 con 0x563a93229760 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fce880d5030 con 0x563a93229760 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fce8809a8a0 con 0x563a93229760 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 10 client.16866 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 10 client.16866 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:12.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 20 client.16866 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a932a0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 4 client.16866 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.618+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x563a93194690 con 0x563a93242780 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fce977fe640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a932a0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fce777fe640 10 client.16866 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fce977fe640 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a932a0c70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fcd6650ccbb2b23e server_cookie=11aed05c9127e407 in_seq=0 out_seq=0 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fce777fe640 1 -- 172.21.15.73:0/995169817 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x563a93194690 con 0x563a93242780 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fce777fe640 10 client.16866 handle_command_reply: tid=0 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 client.16866 shutdown 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 msgr2=0x563a93297200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 crypto rx=0x7fce8809b850 tx=0x7fce8809b8c0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 shutdown_connections 2024-03-09T00:14:12.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x563a93242780 0x563a932a0c70 crc :-1 s=READY pgs=168 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x563a93232c60 0x563a93291610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.619+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x563a9322a080 0x563a93297740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce777fe640 0 client.16866 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fcea1201740 1 --2- 172.21.15.73:0/995169817 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x563a93229760 0x563a93297200 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 >> 172.21.15.73:0/995169817 conn(0x563a93223e90 msgr2=0x563a9322a5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 shutdown_connections 2024-03-09T00:14:12.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fcea1201740 1 -- 172.21.15.73:0/995169817 wait complete. 2024-03-09T00:14:12.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fcea1201740 20 client.16866 trim_cache size 0 max 16384 2024-03-09T00:14:12.634 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (1958 inodes in the stack)", 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "90a86510-42c9-4cf1-9ecc-7ecd22a3bab9": { 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "90a86510-42c9-4cf1-9ecc-7ecd22a3bab9", 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:14:12.635 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:14:12.636 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:14:12.636 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:12.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 msgr2=0x7fce800a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 0x7fce800a4e30 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fce90021000 tx=0x7fce9001e000 comp rx=0 tx=0).stop 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 msgr2=0x7fce9819d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 crypto rx=0x7fce8800e9f0 tx=0x7fce8800edf0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 shutdown_connections 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.620+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fce98197b00 0x7fce98197fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fce981065d0 0x7fce9819de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fce800a2970 0x7fce800a4e30 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 --2- 172.21.15.73:0/1696453110 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fce98101500 0x7fce9819d900 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:12.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 >> 172.21.15.73:0/1696453110 conn(0x7fce980fcd20 msgr2=0x7fce980fe150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:12.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 shutdown_connections 2024-03-09T00:14:12.639 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:12.621+0000 7fce9e620640 1 -- 172.21.15.73:0/1696453110 wait complete. 2024-03-09T00:14:12.639 INFO:tasks.cephfs.filesystem:scrub status for tag:90a86510-42c9-4cf1-9ecc-7ecd22a3bab9 - {'path': '/', 'tag': '90a86510-42c9-4cf1-9ecc-7ecd22a3bab9', 'options': 'recursive,force'} 2024-03-09T00:14:13.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:13 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:14:13.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:13 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:13.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:13 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:14:13.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:13 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:13.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:13 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:14:13.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:13 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:13.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:13 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:14:13.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:13 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:13.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:13 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:14:13.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:13 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:13.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:13 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:14:13.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:13 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:14.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:14 smithi179 ceph-mon[36637]: pgmap v227: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:14:14.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:14 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:14:14.410 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:14 smithi073 ceph-mon[32309]: pgmap v227: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:14:14.410 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:14 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:14:14.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:14 smithi102 ceph-mon[36106]: pgmap v227: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:14:14.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:14 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:14:16.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:15 smithi102 ceph-mon[36106]: osdmap e98: 12 total, 12 up, 12 in 2024-03-09T00:14:16.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:15 smithi102 ceph-mon[36106]: pgmap v229: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:14:16.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:15 smithi179 ceph-mon[36637]: osdmap e98: 12 total, 12 up, 12 in 2024-03-09T00:14:16.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:15 smithi179 ceph-mon[36637]: pgmap v229: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:14:16.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:15 smithi073 ceph-mon[32309]: osdmap e98: 12 total, 12 up, 12 in 2024-03-09T00:14:16.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:15 smithi073 ceph-mon[32309]: pgmap v229: 129 pgs: 129 active+clean; 35 MiB data, 904 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:14:18.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:18 smithi073 ceph-mon[32309]: pgmap v230: 129 pgs: 129 active+clean; 35 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 253 B/s wr, 228 op/s 2024-03-09T00:14:18.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:18 smithi102 ceph-mon[36106]: pgmap v230: 129 pgs: 129 active+clean; 35 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 253 B/s wr, 228 op/s 2024-03-09T00:14:18.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:18 smithi179 ceph-mon[36637]: pgmap v230: 129 pgs: 129 active+clean; 35 MiB data, 888 MiB used, 1.0 TiB / 1.0 TiB avail; 228 KiB/s rd, 253 B/s wr, 228 op/s 2024-03-09T00:14:20.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:20 smithi073 ceph-mon[32309]: pgmap v231: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:20.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:20 smithi102 ceph-mon[36106]: pgmap v231: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:20.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:20 smithi179 ceph-mon[36637]: pgmap v231: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:22.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:22 smithi073 ceph-mon[32309]: pgmap v232: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:22.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:22 smithi102 ceph-mon[36106]: pgmap v232: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:22.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:22 smithi179 ceph-mon[36637]: pgmap v232: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:14:24.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:24 smithi073 ceph-mon[32309]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:14:24.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:24 smithi102 ceph-mon[36106]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:14:24.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:24 smithi179 ceph-mon[36637]: pgmap v233: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:14:26.261 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:25 smithi102 ceph-mon[36106]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:14:26.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:25 smithi179 ceph-mon[36637]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:14:26.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:25 smithi073 ceph-mon[32309]: pgmap v234: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:14:28.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:28 smithi073 ceph-mon[32309]: pgmap v235: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:14:28.511 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:28 smithi102 ceph-mon[36106]: pgmap v235: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:14:28.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:28 smithi179 ceph-mon[36637]: pgmap v235: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:14:30.418 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:30 smithi073 ceph-mon[32309]: pgmap v236: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:14:30.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:30 smithi102 ceph-mon[36106]: pgmap v236: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:14:30.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:30 smithi179 ceph-mon[36637]: pgmap v236: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 136 op/s 2024-03-09T00:14:32.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:32 smithi073 ceph-mon[32309]: pgmap v237: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:32.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:32 smithi102 ceph-mon[36106]: pgmap v237: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:32.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:32 smithi179 ceph-mon[36637]: pgmap v237: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:34.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:34 smithi102 ceph-mon[36106]: pgmap v238: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:34.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:34 smithi179 ceph-mon[36637]: pgmap v238: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:34.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:34 smithi073 ceph-mon[32309]: pgmap v238: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:36.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:36 smithi102 ceph-mon[36106]: pgmap v239: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:36.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:36 smithi179 ceph-mon[36637]: pgmap v239: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:36.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:36 smithi073 ceph-mon[32309]: pgmap v239: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:38.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:38 smithi102 ceph-mon[36106]: pgmap v240: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:38.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:38 smithi179 ceph-mon[36637]: pgmap v240: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:38.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:38 smithi073 ceph-mon[32309]: pgmap v240: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:40.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:40 smithi102 ceph-mon[36106]: pgmap v241: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:40.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:40 smithi179 ceph-mon[36637]: pgmap v241: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:40.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:40 smithi073 ceph-mon[32309]: pgmap v241: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:42.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:42 smithi102 ceph-mon[36106]: pgmap v242: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:42.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:42 smithi179 ceph-mon[36637]: pgmap v242: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:42.640 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:14:42.641 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:14:42.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:42 smithi073 ceph-mon[32309]: pgmap v242: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:42.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 -- 172.21.15.73:0/3517429752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c1074f0 msgr2=0x7f768c1078d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 --2- 172.21.15.73:0/3517429752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c1074f0 0x7f768c1078d0 secure :-1 s=READY pgs=891 cs=0 l=1 rev1=1 crypto rx=0x7f767c009930 tx=0x7f767c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 -- 172.21.15.73:0/3517429752 shutdown_connections 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 --2- 172.21.15.73:0/3517429752 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c100570 0x7f768c10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 --2- 172.21.15.73:0/3517429752 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c0ffbb0 0x7f768c100030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 --2- 172.21.15.73:0/3517429752 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c1074f0 0x7f768c1078d0 unknown :-1 s=CLOSED pgs=891 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 -- 172.21.15.73:0/3517429752 >> 172.21.15.73:0/3517429752 conn(0x7f768c075530 msgr2=0x7f768c075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 -- 172.21.15.73:0/3517429752 shutdown_connections 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.759+0000 7f7691bc4640 1 -- 172.21.15.73:0/3517429752 wait complete. 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 Processor -- start 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 -- start start 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c0ffbb0 0x7f768c100d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 0x7f768c101810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1116d0 con 0x7f768c100570 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c111550 con 0x7f768c1074f0 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f7691bc4640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c111850 con 0x7f768c0ffbb0 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33432/0 (socket says 172.21.15.73:33432) 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.760+0000 7f768affd640 1 -- 172.21.15.73:0/1651669543 learned_addr learned my addr 172.21.15.73:0/1651669543 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768b7fe640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c0ffbb0 0x7f768c100d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768bfff640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 0x7f768c101810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 -- 172.21.15.73:0/1651669543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c0ffbb0 msgr2=0x7f768c100d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c0ffbb0 0x7f768c100d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 -- 172.21.15.73:0/1651669543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 msgr2=0x7f768c101810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 0x7f768c101810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 -- 172.21.15.73:0/1651669543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768c1a6030 con 0x7f768c100570 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768bfff640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 0x7f768c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f768affd640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f768000c920 tx=0x7f768000cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f7688ff9640 1 -- 172.21.15.73:0/1651669543 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680007c70 con 0x7f768c100570 2024-03-09T00:14:42.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768c1a6320 con 0x7f768c100570 2024-03-09T00:14:42.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f7688ff9640 1 -- 172.21.15.73:0/1651669543 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7680007e10 con 0x7f768c100570 2024-03-09T00:14:42.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f7688ff9640 1 -- 172.21.15.73:0/1651669543 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680018890 con 0x7f768c100570 2024-03-09T00:14:42.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.761+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768c1a68c0 con 0x7f768c100570 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f7688ff9640 1 -- 172.21.15.73:0/1651669543 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7680018a70 con 0x7f768c100570 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 Processor -- start 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 -- start start 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bb9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc5050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29b1ce00 con 0x55de29bba080 2024-03-09T00:14:42.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29ac5b10 con 0x55de29bb9760 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f76947a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29b18ea0 con 0x55de29bc2c60 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f7688ff9640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 0x7f76740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f768b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33438/0 (socket says 172.21.15.73:33438) 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.764+0000 7f768b7fe640 1 -- 172.21.15.73:0/879763041 learned_addr learned my addr 172.21.15.73:0/879763041 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768bfff640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 0x7f76740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768affd640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768bfff640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bb9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 -- 172.21.15.73:0/879763041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 msgr2=0x55de29bc5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f7688ff9640 1 -- 172.21.15.73:0/1651669543 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f76800cf430 con 0x7f768c100570 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 -- 172.21.15.73:0/879763041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 msgr2=0x55de29bb9b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bb9b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 -- 172.21.15.73:0/879763041 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de298fb7d0 con 0x55de29bba080 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768bfff640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bb9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:42.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768affd640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768b7fe640 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f767c036040 tx=0x7f767c02c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=318bd639e10c50e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f768bfff640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 0x7f76740a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7684005fd0 tx=0x7f76840094c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f7671ffb640 1 -- 172.21.15.73:0/879763041 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c02cb00 con 0x55de29bba080 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f7671ffb640 1 -- 172.21.15.73:0/879763041 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f767c02cca0 con 0x55de29bba080 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.765+0000 7f7671ffb640 1 -- 172.21.15.73:0/879763041 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f767c0049b0 con 0x55de29bba080 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- 172.21.15.73:0/879763041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 msgr2=0x55de29bc2720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 secure :-1 s=READY pgs=893 cs=0 l=1 rev1=1 crypto rx=0x7f767c036040 tx=0x7f767c02c650 comp rx=0 tx=0).stop 2024-03-09T00:14:42.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- 172.21.15.73:0/879763041 shutdown_connections 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc2720 unknown :-1 s=CLOSED pgs=893 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 --2- 172.21.15.73:0/879763041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bb9b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- 172.21.15.73:0/879763041 >> 172.21.15.73:0/879763041 conn(0x55de29bb3e90 msgr2=0x55de29bb4270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- 172.21.15.73:0/879763041 shutdown_connections 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- 172.21.15.73:0/879763041 wait complete. 2024-03-09T00:14:42.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 -- start start 2024-03-09T00:14:42.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.766+0000 7f76947a1740 1 Processor -- start 2024-03-09T00:14:42.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bc22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bbfc20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29bc59f0 con 0x55de29bba080 2024-03-09T00:14:42.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29ad90a0 con 0x55de29bb9760 2024-03-09T00:14:42.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de29bc5b70 con 0x55de29bc2c60 2024-03-09T00:14:42.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 10 client.0 _ll_register_callbacks cb 0x55de29ad8a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:42.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33448/0 (socket says 172.21.15.73:33448) 2024-03-09T00:14:42.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 -- 172.21.15.73:0/1157228663 learned_addr learned my addr 172.21.15.73:0/1157228663 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768b7fe640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bc22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 -- 172.21.15.73:0/1157228663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 msgr2=0x55de29bbfc20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bbfc20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 -- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 msgr2=0x55de29bc22e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bc22e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 -- 172.21.15.73:0/1157228663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de298313f0 con 0x55de29bba080 2024-03-09T00:14:42.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f768affd640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f768000c8f0 tx=0x7f7680015670 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d0838355123bdbf1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f766affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:42.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7680086070 con 0x55de29bba080 2024-03-09T00:14:42.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f768000fb30 con 0x55de29bba080 2024-03-09T00:14:42.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.767+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f767c02cca0 con 0x55de29bba080 2024-03-09T00:14:42.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768009bda0 con 0x55de29bba080 2024-03-09T00:14:42.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 10 client.16878 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:42.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55de297f10f0 con 0x55de29bba080 2024-03-09T00:14:42.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f767c02cca0 con 0x55de29bba080 2024-03-09T00:14:42.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f76800152b0 con 0x55de29bba080 2024-03-09T00:14:42.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 10 client.16878 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:42.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 10 client.16878 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:42.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 20 client.16878 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:42.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29bd2650 0x55de29bd4a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 4 client.16878 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:42.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de29b2a340 con 0x55de29bd2650 2024-03-09T00:14:42.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.768+0000 7f768b7fe640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29bd2650 0x55de29bd4a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f766affd640 10 client.16878 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f768b7fe640 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29bd2650 0x55de29bd4a40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d15357ec283e062 server_cookie=110496e8c48d2d76 in_seq=0 out_seq=0 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f766affd640 1 -- 172.21.15.73:0/1157228663 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55de29b2a340 con 0x55de29bd2650 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f766affd640 10 client.16878 handle_command_reply: tid=0 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 client.16878 shutdown 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 msgr2=0x55de29bbf6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 crypto rx=0x7f768000c8f0 tx=0x7f7680015670 comp rx=0 tx=0).stop 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 shutdown_connections 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29bd2650 0x55de29bd4a40 crc :-1 s=READY pgs=169 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bbfc20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bbf6e0 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f766affd640 0 client.16878 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.769+0000 7f76947a1740 1 --2- 172.21.15.73:0/1157228663 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bc22e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.770+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 >> 172.21.15.73:0/1157228663 conn(0x55de29bb3e90 msgr2=0x55de29bb5be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.789 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.770+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 shutdown_connections 2024-03-09T00:14:42.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.770+0000 7f76947a1740 1 -- 172.21.15.73:0/1157228663 wait complete. 2024-03-09T00:14:42.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.770+0000 7f76947a1740 20 client.16878 trim_cache size 0 max 16384 2024-03-09T00:14:42.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.784+0000 7f76947a1740 1 Processor -- start 2024-03-09T00:14:42.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.784+0000 7f76947a1740 1 -- start start 2024-03-09T00:14:42.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bbefc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 0x55de29bc9da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1ae950 con 0x55de29bc2c60 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1b0ef0 con 0x55de29bb9760 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76947a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1aef50 con 0x55de29bba080 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33462/0 (socket says 172.21.15.73:33462) 2024-03-09T00:14:42.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 -- 172.21.15.73:0/2967708258 learned_addr learned my addr 172.21.15.73:0/2967708258 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768affd640 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 0x55de29bc9da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 -- 172.21.15.73:0/2967708258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 msgr2=0x55de29bc9da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 0x55de29bc9da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 -- 172.21.15.73:0/2967708258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 msgr2=0x55de29bbefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bbefc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 -- 172.21.15.73:0/2967708258 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de29bb8850 con 0x55de29bc2c60 2024-03-09T00:14:42.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768affd640 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 0x55de29bc9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f768bfff640 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f76840125e0 tx=0x7f7684009240 comp rx=0 tx=0).ready entity=mon.0 client_cookie=436fd3b0b7a7ce4a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76717fa640 1 -- 172.21.15.73:0/2967708258 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7684016070 con 0x55de29bc2c60 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76717fa640 1 -- 172.21.15.73:0/2967708258 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7684025ae0 con 0x55de29bc2c60 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.785+0000 7f76717fa640 1 -- 172.21.15.73:0/2967708258 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f768401dcf0 con 0x55de29bc2c60 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- 172.21.15.73:0/2967708258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 msgr2=0x55de29bca2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 secure :-1 s=READY pgs=895 cs=0 l=1 rev1=1 crypto rx=0x7f76840125e0 tx=0x7f7684009240 comp rx=0 tx=0).stop 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- 172.21.15.73:0/2967708258 shutdown_connections 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bc2c60 0x55de29bca2e0 unknown :-1 s=CLOSED pgs=895 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bba080 0x55de29bc9da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 --2- 172.21.15.73:0/2967708258 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bbefc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- 172.21.15.73:0/2967708258 >> 172.21.15.73:0/2967708258 conn(0x55de29bb3e90 msgr2=0x55de29bbf6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- 172.21.15.73:0/2967708258 shutdown_connections 2024-03-09T00:14:42.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- 172.21.15.73:0/2967708258 wait complete. 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 -- start start 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.786+0000 7f76947a1740 1 Processor -- start 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bca110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc6980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1af570 con 0x55de29bba080 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1b0890 con 0x55de29bb9760 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768c1afbd0 con 0x55de29bc2c60 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 10 client.0 _ll_register_callbacks cb 0x7f768000fe30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33476/0 (socket says 172.21.15.73:33476) 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 -- 172.21.15.73:0/2480971222 learned_addr learned my addr 172.21.15.73:0/2480971222 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768b7fe640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bca110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768bfff640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc6980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 -- 172.21.15.73:0/2480971222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 msgr2=0x55de29bc6980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc6980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 -- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 msgr2=0x55de29bca110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bca110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 -- 172.21.15.73:0/2480971222 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de29bc7190 con 0x55de29bba080 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768bfff640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f768affd640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f76800cc840 tx=0x7f7680020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be5d319637532976 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f766affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76800b9060 con 0x55de29bba080 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f76800152b0 con 0x55de29bba080 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7684025ae0 con 0x55de29bba080 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.787+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f768009b470 con 0x55de29bba080 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 10 client.16884 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:42.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55de29bc7190 con 0x55de29bba080 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7684025ae0 con 0x55de29bba080 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7680060ce0 con 0x55de29bba080 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 10 client.16884 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 10 client.16884 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 20 client.16884 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29c29df0 0x55de29c2c210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 4 client.16884 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:14:42.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55de29b24690 con 0x55de29c29df0 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f768b7fe640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29c29df0 0x55de29c2c210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f766affd640 10 client.16884 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.788+0000 7f768b7fe640 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29c29df0 0x55de29c2c210 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f196fc1d98abc2f9 server_cookie=400f2aa547360b00 in_seq=0 out_seq=0 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f766affd640 1 -- 172.21.15.73:0/2480971222 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55de29b24690 con 0x55de29c29df0 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f766affd640 10 client.16884 handle_command_reply: tid=0 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 client.16884 shutdown 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 msgr2=0x55de29bc6440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 crypto rx=0x7f76800cc840 tx=0x7f7680020000 comp rx=0 tx=0).stop 2024-03-09T00:14:42.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 shutdown_connections 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55de29c29df0 0x55de29c2c210 crc :-1 s=READY pgs=170 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55de29bc2c60 0x55de29bc6980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f766affd640 0 client.16884 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55de29bba080 0x55de29bc6440 unknown :-1 s=CLOSED pgs=896 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 --2- 172.21.15.73:0/2480971222 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55de29bb9760 0x55de29bca110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 >> 172.21.15.73:0/2480971222 conn(0x55de29bb3e90 msgr2=0x55de29bba5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 shutdown_connections 2024-03-09T00:14:42.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 1 -- 172.21.15.73:0/2480971222 wait complete. 2024-03-09T00:14:42.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f76947a1740 20 client.16884 trim_cache size 0 max 16384 2024-03-09T00:14:42.811 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:42.811 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:14:42.811 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:14:42.811 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:42.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 msgr2=0x7f76740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 0x7f76740a4e30 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f7684005fd0 tx=0x7f76840094c0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 msgr2=0x7f768c1012d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.789+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 crypto rx=0x7f768000c920 tx=0x7f768000cde0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 shutdown_connections 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f768c1074f0 0x7f768c101810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f76740a2970 0x7f76740a4e30 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f768c100570 0x7f768c1012d0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 --2- 172.21.15.73:0/1651669543 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f768c0ffbb0 0x7f768c100d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 >> 172.21.15.73:0/1651669543 conn(0x7f768c075530 msgr2=0x7f768c0fe600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 shutdown_connections 2024-03-09T00:14:42.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.790+0000 7f7691bc4640 1 -- 172.21.15.73:0/1651669543 wait complete. 2024-03-09T00:14:42.815 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:14:42.815 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:14:42.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 -- 172.21.15.73:0/4190135717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 msgr2=0x7f27a410dda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 --2- 172.21.15.73:0/4190135717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a410dda0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 crypto rx=0x7f27a0009900 tx=0x7f27a002bcd0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 -- 172.21.15.73:0/4190135717 shutdown_connections 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 --2- 172.21.15.73:0/4190135717 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a410dda0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 --2- 172.21.15.73:0/4190135717 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102ea0 0x7f27a410b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 --2- 172.21.15.73:0/4190135717 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a4102580 0x7f27a4102960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 -- 172.21.15.73:0/4190135717 >> 172.21.15.73:0/4190135717 conn(0x7f27a4077350 msgr2=0x7f27a4077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 -- 172.21.15.73:0/4190135717 shutdown_connections 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.957+0000 7f27ab481640 1 -- 172.21.15.73:0/4190135717 wait complete. 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 Processor -- start 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 -- start start 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102580 0x7f27a41001d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a4102ea0 0x7f27a4100710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a4111740 con 0x7f27a410b950 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41115c0 con 0x7f27a4102ea0 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27ab481640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27a41118c0 con 0x7f27a4102580 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33494/0 (socket says 172.21.15.73:33494) 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 -- 172.21.15.73:0/371607063 learned_addr learned my addr 172.21.15.73:0/371607063 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:42.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a921d640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102580 0x7f27a41001d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 -- 172.21.15.73:0/371607063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102580 msgr2=0x7f27a41001d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102580 0x7f27a41001d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 -- 172.21.15.73:0/371607063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a4102ea0 msgr2=0x7f27a4100710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a4102ea0 0x7f27a4100710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.958+0000 7f27a9a1e640 1 -- 172.21.15.73:0/371607063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27a41a60a0 con 0x7f27a410b950 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f27a9a1e640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f27a0005940 tx=0x7f27a002cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0041070 con 0x7f27a410b950 2024-03-09T00:14:42.961 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f27a00054e0 con 0x7f27a410b950 2024-03-09T00:14:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27a41a6330 con 0x7f27a410b950 2024-03-09T00:14:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f27a0003e50 con 0x7f27a410b950 2024-03-09T00:14:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.959+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27a41a6820 con 0x7f27a410b950 2024-03-09T00:14:42.962 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.960+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27a4068530 con 0x7f27a410b950 2024-03-09T00:14:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.961+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f27a0044070 con 0x7f27a410b950 2024-03-09T00:14:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.961+0000 7f279a7fc640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 0x7f27900a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.962+0000 7f27a921d640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 0x7f27900a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.962+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f27a00f0e10 con 0x7f27a410b950 2024-03-09T00:14:42.963 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.962+0000 7f27a921d640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 0x7f27900a5000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f279c00a400 tx=0x7f279c00d040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:42.964 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:42.966+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f27a00bd890 con 0x7f27a410b950 2024-03-09T00:14:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.177+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f27a41a6a80 con 0x7f27a410b950 2024-03-09T00:14:43.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.178+0000 7f279a7fc640 1 -- 172.21.15.73:0/371607063 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f27a00bd190 con 0x7f27a410b950 2024-03-09T00:14:43.176 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:14:43.176 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:14:43.176 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:14:43.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.179+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 msgr2=0x7f27900a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.179+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 0x7f27900a5000 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f279c00a400 tx=0x7f279c00d040 comp rx=0 tx=0).stop 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.179+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 msgr2=0x7f27a4100c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.179+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 secure :-1 s=READY pgs=898 cs=0 l=1 rev1=1 crypto rx=0x7f27a0005940 tx=0x7f27a002cd10 comp rx=0 tx=0).stop 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 shutdown_connections 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f27900a2b40 0x7f27900a5000 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f27a410b950 0x7f27a4100c50 unknown :-1 s=CLOSED pgs=898 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f27a4102ea0 0x7f27a4100710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 --2- 172.21.15.73:0/371607063 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f27a4102580 0x7f27a41001d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 >> 172.21.15.73:0/371607063 conn(0x7f27a4077350 msgr2=0x7f27a40fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 shutdown_connections 2024-03-09T00:14:43.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.180+0000 7f27ab481640 1 -- 172.21.15.73:0/371607063 wait complete. 2024-03-09T00:14:43.188 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:14:43.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:14:43.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 -- 172.21.15.73:0/3419458685 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c100d90 msgr2=0x7fbf1c104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 --2- 172.21.15.73:0/3419458685 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c100d90 0x7fbf1c104dd0 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 crypto rx=0x7fbf10009900 tx=0x7fbf1002bc30 comp rx=0 tx=0).stop 2024-03-09T00:14:43.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 -- 172.21.15.73:0/3419458685 shutdown_connections 2024-03-09T00:14:43.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 --2- 172.21.15.73:0/3419458685 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 --2- 172.21.15.73:0/3419458685 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c100d90 0x7fbf1c104dd0 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 --2- 172.21.15.73:0/3419458685 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 0x7fbf1c1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.294+0000 7fbf20c59640 1 -- 172.21.15.73:0/3419458685 >> 172.21.15.73:0/3419458685 conn(0x7fbf1c0773a0 msgr2=0x7fbf1c0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- 172.21.15.73:0/3419458685 shutdown_connections 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- 172.21.15.73:0/3419458685 wait complete. 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 Processor -- start 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- start start 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 0x7fbf1c19d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 0x7fbf1c197da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1105a0 con 0x7fbf1c1054b0 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c110420 con 0x7fbf1c1003e0 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.295+0000 7fbf20c59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c110720 con 0x7fbf1c1978f0 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33504/0 (socket says 172.21.15.73:33504) 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 -- 172.21.15.73:0/167395926 learned_addr learned my addr 172.21.15.73:0/167395926 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 0x7fbf1c197da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 -- 172.21.15.73:0/167395926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 msgr2=0x7fbf1c197da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 0x7fbf1c197da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 -- 172.21.15.73:0/167395926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 msgr2=0x7fbf1c19d740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 0x7fbf1c19d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 0x7fbf1c19d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 -- 172.21.15.73:0/167395926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf1c198410 con 0x7fbf1c1054b0 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 0x7fbf1c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbf10000c00 tx=0x7fbf1002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf0b7fe640 1 -- 172.21.15.73:0/167395926 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf10041070 con 0x7fbf1c1054b0 2024-03-09T00:14:43.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf1c1986a0 con 0x7fbf1c1054b0 2024-03-09T00:14:43.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf0b7fe640 1 -- 172.21.15.73:0/167395926 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf10004450 con 0x7fbf1c1054b0 2024-03-09T00:14:43.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf0b7fe640 1 -- 172.21.15.73:0/167395926 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf10034440 con 0x7fbf1c1054b0 2024-03-09T00:14:43.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.296+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf1c1a4890 con 0x7fbf1c1054b0 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.298+0000 7fbf0b7fe640 1 -- 172.21.15.73:0/167395926 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbf10044070 con 0x7fbf1c1054b0 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.298+0000 7fbf23837740 1 Processor -- start 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 -- start start 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf0b7fe640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 0x7fbf040a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf38230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf45ab0 0x55f54bf47ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54beff360 con 0x55f54bf38770 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54be9bec0 con 0x55f54bf45ab0 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf23837740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54be44d30 con 0x55f54bf37e50 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 0x7fbf040a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf0b7fe640 1 -- 172.21.15.73:0/167395926 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fbf100f0ef0 con 0x7fbf1c1054b0 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33518/0 (socket says 172.21.15.73:33518) 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 -- 172.21.15.73:0/933194766 learned_addr learned my addr 172.21.15.73:0/933194766 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf38230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf45ab0 0x55f54bf47ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 -- 172.21.15.73:0/933194766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 msgr2=0x55f54bf38230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf38230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 -- 172.21.15.73:0/933194766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf45ab0 msgr2=0x55f54bf47ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf45ab0 0x55f54bf47ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf19d9b640 1 -- 172.21.15.73:0/933194766 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f54be43bc0 con 0x55f54bf38770 2024-03-09T00:14:43.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf38230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.299+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 0x7fbf040a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf14006fd0 tx=0x7fbf14013650 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbf100b9c40 tx=0x7fbf100bce70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7c7c7207aa54d640 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf08ff9640 1 -- 172.21.15.73:0/933194766 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf1004b070 con 0x55f54bf38770 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf08ff9640 1 -- 172.21.15.73:0/933194766 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf100bd970 con 0x55f54bf38770 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf08ff9640 1 -- 172.21.15.73:0/933194766 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf100c5a10 con 0x55f54bf38770 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 -- 172.21.15.73:0/933194766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 msgr2=0x55f54bf45570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 secure :-1 s=READY pgs=901 cs=0 l=1 rev1=1 crypto rx=0x7fbf100b9c40 tx=0x7fbf100bce70 comp rx=0 tx=0).stop 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 -- 172.21.15.73:0/933194766 shutdown_connections 2024-03-09T00:14:43.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf45ab0 0x55f54bf47ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf38770 0x55f54bf45570 unknown :-1 s=CLOSED pgs=901 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 --2- 172.21.15.73:0/933194766 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf38230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 -- 172.21.15.73:0/933194766 >> 172.21.15.73:0/933194766 conn(0x55f54bf328d0 msgr2=0x55f54bf32cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 -- 172.21.15.73:0/933194766 shutdown_connections 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.300+0000 7fbf23837740 1 -- 172.21.15.73:0/933194766 wait complete. 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 -- start start 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 Processor -- start 2024-03-09T00:14:43.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf42b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf43050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54be3a4a0 con 0x55f54bf45ab0 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54be97f60 con 0x55f54bf38770 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f54bf486b0 con 0x55f54bf37e50 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf23837740 10 client.0 _ll_register_callbacks cb 0x55f54be95d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33532/0 (socket says 172.21.15.73:33532) 2024-03-09T00:14:43.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/168670497 learned_addr learned my addr 172.21.15.73:0/168670497 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf42b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf43050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/168670497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 msgr2=0x55f54bf42b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf42b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 msgr2=0x55f54bf43050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf43050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/168670497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f54bbb03f0 con 0x55f54bf45ab0 2024-03-09T00:14:43.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf42b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.301+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf43050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf14018b80 tx=0x7fbf140046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fa0c5a2599b7539c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14027060 con 0x55f54bf45ab0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fbf14015860 con 0x55f54bf45ab0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf100bd970 con 0x55f54bf45ab0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf1402e050 con 0x55f54bf45ab0 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 10 client.16902 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:43.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f54bc7a7d0 con 0x55f54bf45ab0 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf100bd970 con 0x55f54bf45ab0 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf14015040 con 0x55f54bf45ab0 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 10 client.16902 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 10 client.16902 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 20 client.16902 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bf55a70 0x55f54bf57e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 4 client.16902 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:43.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.302+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f54bea9400 con 0x55f54bf55a70 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.303+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bf55a70 0x55f54bf57e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.303+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bf55a70 0x55f54bf57e60 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4e2087593adece8 server_cookie=8b4ba7b7fab3fd9d in_seq=0 out_seq=0 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.303+0000 7fbf01ffb640 10 client.16902 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf01ffb640 1 -- 172.21.15.73:0/168670497 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f54bea9400 con 0x55f54bf55a70 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf01ffb640 10 client.16902 handle_command_reply: tid=0 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 client.16902 shutdown 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 msgr2=0x55f54bf43590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 secure :-1 s=READY pgs=902 cs=0 l=1 rev1=1 crypto rx=0x7fbf14018b80 tx=0x7fbf140046c0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 shutdown_connections 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bf55a70 0x55f54bf57e60 crc :-1 s=READY pgs=171 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bf43590 unknown :-1 s=CLOSED pgs=902 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf43050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 --2- 172.21.15.73:0/168670497 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf42b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf01ffb640 0 client.16902 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 >> 172.21.15.73:0/168670497 conn(0x55f54bf328d0 msgr2=0x55f54bf34620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 shutdown_connections 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 1 -- 172.21.15.73:0/168670497 wait complete. 2024-03-09T00:14:43.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.304+0000 7fbf23837740 20 client.16902 trim_cache size 0 max 16384 2024-03-09T00:14:43.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 Processor -- start 2024-03-09T00:14:43.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 -- start start 2024-03-09T00:14:43.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf3fd70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.323 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf45ab0 0x55f54bf402b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1aa5b0 con 0x55f54bf37e50 2024-03-09T00:14:43.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1acb50 con 0x55f54bf38770 2024-03-09T00:14:43.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.315+0000 7fbf23837740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1aabb0 con 0x55f54bf45ab0 2024-03-09T00:14:43.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.324 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33536/0 (socket says 172.21.15.73:33536) 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 -- 172.21.15.73:0/1985826320 learned_addr learned my addr 172.21.15.73:0/1985826320 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf45ab0 0x55f54bf402b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 -- 172.21.15.73:0/1985826320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf45ab0 msgr2=0x55f54bf402b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf45ab0 0x55f54bf402b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 -- 172.21.15.73:0/1985826320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 msgr2=0x55f54bf3fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf3fd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 -- 172.21.15.73:0/1985826320 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f54bf581d0 con 0x55f54bf37e50 2024-03-09T00:14:43.325 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c008500 tx=0x7fbf0c008900 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8fc9eda764fdc7f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf03fff640 1 -- 172.21.15.73:0/1985826320 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c011b70 con 0x55f54bf37e50 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf03fff640 1 -- 172.21.15.73:0/1985826320 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf0c011d10 con 0x55f54bf37e50 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.316+0000 7fbf03fff640 1 -- 172.21.15.73:0/1985826320 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf0c005670 con 0x55f54bf37e50 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- 172.21.15.73:0/1985826320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 msgr2=0x55f54bf45240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 secure :-1 s=READY pgs=903 cs=0 l=1 rev1=1 crypto rx=0x7fbf0c008500 tx=0x7fbf0c008900 comp rx=0 tx=0).stop 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- 172.21.15.73:0/1985826320 shutdown_connections 2024-03-09T00:14:43.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf45ab0 0x55f54bf402b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf3fd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- 172.21.15.73:0/1985826320 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf37e50 0x55f54bf45240 unknown :-1 s=CLOSED pgs=903 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- 172.21.15.73:0/1985826320 >> 172.21.15.73:0/1985826320 conn(0x55f54bf328d0 msgr2=0x55f54bf413e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- 172.21.15.73:0/1985826320 shutdown_connections 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- 172.21.15.73:0/1985826320 wait complete. 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- start start 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 Processor -- start 2024-03-09T00:14:43.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf4bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf4c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1ab1d0 con 0x55f54bf45ab0 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1ac4f0 con 0x55f54bf38770 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf1c1ab830 con 0x55f54bf37e50 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf23837740 10 client.0 _ll_register_callbacks cb 0x7fbf10009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf1ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf1ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33548/0 (socket says 172.21.15.73:33548) 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/2319628501 learned_addr learned my addr 172.21.15.73:0/2319628501 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.317+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf4bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf19d9b640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf4c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/2319628501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 msgr2=0x55f54bf4bb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf4bb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 msgr2=0x55f54bf4c060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf4c060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 -- 172.21.15.73:0/2319628501 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f54bfac4d0 con 0x55f54bf45ab0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf1ad9d640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf140228a0 tx=0x7fbf14022dc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b43494a87f47f83e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbf14027070 con 0x55f54bf45ab0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fbf1400aa40 con 0x55f54bf45ab0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbf0c011d10 con 0x55f54bf45ab0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbf1400f040 con 0x55f54bf45ab0 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 10 client.16908 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:43.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f54bfad7d0 con 0x55f54bf45ab0 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbf0c011d10 con 0x55f54bf45ab0 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fbf14015040 con 0x55f54bf45ab0 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 10 client.16908 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 10 client.16908 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 20 client.16908 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bfaddd0 0x55f54bfb01f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 4 client.16908 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.318+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f54bea3750 con 0x55f54bfaddd0 2024-03-09T00:14:43.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bfaddd0 0x55f54bfb01f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf1a59c640 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bfaddd0 0x55f54bfb01f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fa32acdc413844e server_cookie=81c4800510519f8 in_seq=0 out_seq=0 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf01ffb640 10 client.16908 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf01ffb640 1 -- 172.21.15.73:0/2319628501 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f54bea3750 con 0x55f54bfaddd0 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf01ffb640 10 client.16908 handle_command_reply: tid=0 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 client.16908 shutdown 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 msgr2=0x55f54bfabf10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 secure :-1 s=READY pgs=904 cs=0 l=1 rev1=1 crypto rx=0x7fbf140228a0 tx=0x7fbf14022dc0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 shutdown_connections 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f54bfaddd0 0x55f54bfb01f0 crc :-1 s=READY pgs=172 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f54bf45ab0 0x55f54bfabf10 unknown :-1 s=CLOSED pgs=904 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f54bf38770 0x55f54bf4c060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf01ffb640 0 client.16908 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 --2- 172.21.15.73:0/2319628501 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f54bf37e50 0x55f54bf4bb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 >> 172.21.15.73:0/2319628501 conn(0x55f54bf328d0 msgr2=0x55f54bf33d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 shutdown_connections 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 1 -- 172.21.15.73:0/2319628501 wait complete. 2024-03-09T00:14:43.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.319+0000 7fbf23837740 20 client.16908 trim_cache size 0 max 16384 2024-03-09T00:14:43.334 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:14:43.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 msgr2=0x7fbf040a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 0x7fbf040a4e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbf14006fd0 tx=0x7fbf14013650 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 msgr2=0x7fbf1c19dc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 crypto rx=0x7fbf10000c00 tx=0x7fbf1002cc70 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 shutdown_connections 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbf1c1978f0 0x7fbf1c197da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbf040a2970 0x7fbf040a4e30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbf1c1054b0 0x7fbf1c19dc80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 --2- 172.21.15.73:0/167395926 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbf1c1003e0 0x7fbf1c19d740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:43.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 >> 172.21.15.73:0/167395926 conn(0x7fbf1c0773a0 msgr2=0x7fbf1c0685f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:43.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 shutdown_connections 2024-03-09T00:14:43.337 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:43.320+0000 7fbf20c59640 1 -- 172.21.15.73:0/167395926 wait complete. 2024-03-09T00:14:44.338 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:14:44.338 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:14:44.339 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:14:44.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 -- 172.21.15.73:0/1013693759 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c100f30 msgr2=0x7f806c101310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 --2- 172.21.15.73:0/1013693759 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c100f30 0x7f806c101310 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 crypto rx=0x7f8054009900 tx=0x7f805402bc30 comp rx=0 tx=0).stop 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 -- 172.21.15.73:0/1013693759 shutdown_connections 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 --2- 172.21.15.73:0/1013693759 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 0x7f806c10d0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 --2- 172.21.15.73:0/1013693759 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c1018e0 0x7f806c104e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 --2- 172.21.15.73:0/1013693759 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c100f30 0x7f806c101310 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 -- 172.21.15.73:0/1013693759 >> 172.21.15.73:0/1013693759 conn(0x7f806c075530 msgr2=0x7f806c075940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 -- 172.21.15.73:0/1013693759 shutdown_connections 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.458+0000 7f8071c30640 1 -- 172.21.15.73:0/1013693759 wait complete. 2024-03-09T00:14:44.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 Processor -- start 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 -- start start 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c100f30 0x7f806c199db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 0x7f806c19a2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c110630 con 0x7f806c19e620 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1104b0 con 0x7f806c100f30 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f8071c30640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1107b0 con 0x7f806c1055b0 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f806bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f806bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33562/0 (socket says 172.21.15.73:33562) 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f806bfff640 1 -- 172.21.15.73:0/3860778007 learned_addr learned my addr 172.21.15.73:0/3860778007 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.459+0000 7f806b7fe640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c100f30 0x7f806c199db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806affd640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 0x7f806c19a2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 -- 172.21.15.73:0/3860778007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 msgr2=0x7f806c19a2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 0x7f806c19a2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 -- 172.21.15.73:0/3860778007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c100f30 msgr2=0x7f806c199db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c100f30 0x7f806c199db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 -- 172.21.15.73:0/3860778007 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806c19f110 con 0x7f806c19e620 2024-03-09T00:14:44.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f806bfff640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f806400b970 tx=0x7f806400be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f8068ff9640 1 -- 172.21.15.73:0/3860778007 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064019070 con 0x7f806c19e620 2024-03-09T00:14:44.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f806c19f400 con 0x7f806c19e620 2024-03-09T00:14:44.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f8068ff9640 1 -- 172.21.15.73:0/3860778007 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f806400cb40 con 0x7f806c19e620 2024-03-09T00:14:44.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f8068ff9640 1 -- 172.21.15.73:0/3860778007 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8064007650 con 0x7f806c19e620 2024-03-09T00:14:44.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.460+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f806c1a6bf0 con 0x7f806c19e620 2024-03-09T00:14:44.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.462+0000 7f8068ff9640 1 -- 172.21.15.73:0/3860778007 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f806401c070 con 0x7f806c19e620 2024-03-09T00:14:44.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.462+0000 7f807480c740 1 Processor -- start 2024-03-09T00:14:44.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 -- start start 2024-03-09T00:14:44.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f8068ff9640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 0x7f80580a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16da670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f1632b40 con 0x55c1f16ced80 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f15dbd80 con 0x55c1f16cf6a0 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f807480c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f162ebe0 con 0x55c1f16d8280 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806affd640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 0x7f80580a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f8068ff9640 1 -- 172.21.15.73:0/3860778007 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f80640ce6b0 con 0x7f806c19e620 2024-03-09T00:14:44.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33566/0 (socket says 172.21.15.73:33566) 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 -- 172.21.15.73:0/3766185713 learned_addr learned my addr 172.21.15.73:0/3766185713 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 -- 172.21.15.73:0/3766185713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 msgr2=0x55c1f16da670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16da670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 -- 172.21.15.73:0/3766185713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 msgr2=0x55c1f16d7d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 -- 172.21.15.73:0/3766185713 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1f140f7e0 con 0x55c1f16ced80 2024-03-09T00:14:44.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806affd640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 0x7f80580a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f805c005d90 tx=0x7f805c00a5a0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f806b7fe640 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f805402cfd0 tx=0x7f805402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aac4ec54a1de4a48 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.463+0000 7f8061ffb640 1 -- 172.21.15.73:0/3766185713 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054042070 con 0x55c1f16ced80 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f8061ffb640 1 -- 172.21.15.73:0/3766185713 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f805402cb40 con 0x55c1f16ced80 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f8061ffb640 1 -- 172.21.15.73:0/3766185713 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8054005200 con 0x55c1f16ced80 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 -- 172.21.15.73:0/3766185713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 msgr2=0x55c1f16cf160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 crypto rx=0x7f805402cfd0 tx=0x7f805402c650 comp rx=0 tx=0).stop 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 -- 172.21.15.73:0/3766185713 shutdown_connections 2024-03-09T00:14:44.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16da670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d7d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 --2- 172.21.15.73:0/3766185713 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16ced80 0x55c1f16cf160 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 -- 172.21.15.73:0/3766185713 >> 172.21.15.73:0/3766185713 conn(0x55c1f16c94b0 msgr2=0x55c1f16c9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 -- 172.21.15.73:0/3766185713 shutdown_connections 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.464+0000 7f807480c740 1 -- 172.21.15.73:0/3766185713 wait complete. 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 -- start start 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 Processor -- start 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16d7d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16d8280 0x55c1f16d5800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f16db010 con 0x55c1f16cf6a0 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f15ef1c0 con 0x55c1f16d8280 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c1f16db190 con 0x55c1f16ced80 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f807480c740 10 client.0 _ll_register_callbacks cb 0x55c1f15de7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33576/0 (socket says 172.21.15.73:33576) 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 -- 172.21.15.73:0/2207636359 learned_addr learned my addr 172.21.15.73:0/2207636359 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16d7d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806bfff640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16d8280 0x55c1f16d5800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 -- 172.21.15.73:0/2207636359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 msgr2=0x55c1f16d7d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16d7d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 -- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16d8280 msgr2=0x55c1f16d5800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16d8280 0x55c1f16d5800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.465+0000 7f806affd640 1 -- 172.21.15.73:0/2207636359 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1f146ff70 con 0x55c1f16cf6a0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f806affd640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f805c00a360 tx=0x7f805c0046c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0628ba80a20377 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c023030 con 0x55c1f16cf6a0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f805c011b40 con 0x55c1f16cf6a0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f805402cb40 con 0x55c1f16cf6a0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f805c02d050 con 0x55c1f16cf6a0 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 10 client.16920 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:44.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c1f1307170 con 0x55c1f16cf6a0 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f805402cb40 con 0x55c1f16cf6a0 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f805c004aa0 con 0x55c1f16cf6a0 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 10 client.16920 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 10 client.16920 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 20 client.16920 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f16e8510 0x55c1f16ea900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 4 client.16920 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:44.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.466+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c1f1608960 con 0x55c1f16e8510 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.467+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f16e8510 0x55c1f16ea900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.467+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f16e8510 0x55c1f16ea900 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=78933af131ba623c server_cookie=ed5b7cf26d65fb4 in_seq=0 out_seq=0 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.467+0000 7f804affd640 10 client.16920 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f804affd640 1 -- 172.21.15.73:0/2207636359 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55c1f1608960 con 0x55c1f16e8510 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f804affd640 10 client.16920 handle_command_reply: tid=0 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 client.16920 shutdown 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 msgr2=0x55c1f16d50c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 crypto rx=0x7f805c00a360 tx=0x7f805c0046c0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 shutdown_connections 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f16e8510 0x55c1f16ea900 crc :-1 s=READY pgs=173 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16d8280 0x55c1f16d5800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16d50c0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f804affd640 0 client.16920 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 --2- 172.21.15.73:0/2207636359 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16d7d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 >> 172.21.15.73:0/2207636359 conn(0x55c1f16c94b0 msgr2=0x55c1f16cb200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 shutdown_connections 2024-03-09T00:14:44.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 1 -- 172.21.15.73:0/2207636359 wait complete. 2024-03-09T00:14:44.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.468+0000 7f807480c740 20 client.16920 trim_cache size 0 max 16384 2024-03-09T00:14:44.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 Processor -- start 2024-03-09T00:14:44.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 -- start start 2024-03-09T00:14:44.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16ced80 0x55c1f16e4870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.487 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16df970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1ae8f0 con 0x55c1f16cf6a0 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1b0e90 con 0x55c1f16ced80 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f807480c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1aeef0 con 0x55c1f16d8280 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33590/0 (socket says 172.21.15.73:33590) 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 -- 172.21.15.73:0/579736402 learned_addr learned my addr 172.21.15.73:0/579736402 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.488 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806bfff640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16df970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 -- 172.21.15.73:0/579736402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 msgr2=0x55c1f16df970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16df970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 -- 172.21.15.73:0/579736402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16ced80 msgr2=0x55c1f16e4870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806b7fe640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16ced80 0x55c1f16e4870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16ced80 0x55c1f16e4870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.483+0000 7f806affd640 1 -- 172.21.15.73:0/579736402 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c1f16d5b70 con 0x55c1f16cf6a0 2024-03-09T00:14:44.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f806bfff640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16df970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f806affd640 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f805c004ec0 tx=0x7f805c012660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6983490aae82d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f80617fa640 1 -- 172.21.15.73:0/579736402 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c023040 con 0x55c1f16cf6a0 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f80617fa640 1 -- 172.21.15.73:0/579736402 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f805c035070 con 0x55c1f16cf6a0 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f80617fa640 1 -- 172.21.15.73:0/579736402 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f805c01f900 con 0x55c1f16cf6a0 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 -- 172.21.15.73:0/579736402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 msgr2=0x55c1f16df430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 crypto rx=0x7f805c004ec0 tx=0x7f805c012660 comp rx=0 tx=0).stop 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 -- 172.21.15.73:0/579736402 shutdown_connections 2024-03-09T00:14:44.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16d8280 0x55c1f16df970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16cf6a0 0x55c1f16df430 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 --2- 172.21.15.73:0/579736402 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16ced80 0x55c1f16e4870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 -- 172.21.15.73:0/579736402 >> 172.21.15.73:0/579736402 conn(0x55c1f16c94b0 msgr2=0x55c1f16d0380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 -- 172.21.15.73:0/579736402 shutdown_connections 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.484+0000 7f807480c740 1 -- 172.21.15.73:0/579736402 wait complete. 2024-03-09T00:14:44.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 -- start start 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 Processor -- start 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16de770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16decb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1af510 con 0x55c1f16d8280 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1b0830 con 0x55c1f16cf6a0 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806c1afb70 con 0x55c1f16ced80 2024-03-09T00:14:44.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f807480c740 10 client.0 _ll_register_callbacks cb 0x7f8054005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f806bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f806bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33606/0 (socket says 172.21.15.73:33606) 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f806bfff640 1 -- 172.21.15.73:0/2669160686 learned_addr learned my addr 172.21.15.73:0/2669160686 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.485+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16de770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806affd640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16decb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 -- 172.21.15.73:0/2669160686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 msgr2=0x55c1f16de770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16de770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 -- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 msgr2=0x55c1f16decb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16decb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 -- 172.21.15.73:0/2669160686 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c1f16cf4d0 con 0x55c1f16d8280 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f806bfff640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f806409aa70 tx=0x7f806409aba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=73d4eeb671b62350 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80640b2070 con 0x55c1f16d8280 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f8064097150 con 0x55c1f16d8280 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f805c035070 con 0x55c1f16d8280 2024-03-09T00:14:44.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80640b92b0 con 0x55c1f16d8280 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 10 client.16926 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55c1f17402a0 con 0x55c1f16d8280 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f805c035070 con 0x55c1f16d8280 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f8064099f30 con 0x55c1f16d8280 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 10 client.16926 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 10 client.16926 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 20 client.16926 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:44.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.486+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f1740970 0x55c1f1742d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.487+0000 7f807480c740 4 client.16926 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.487+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c1f15f4080 con 0x55c1f1740970 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.487+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f1740970 0x55c1f1742d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.487+0000 7f804affd640 10 client.16926 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.487+0000 7f806b7fe640 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f1740970 0x55c1f1742d90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1545be84d6a778eb server_cookie=2b324c5339993899 in_seq=0 out_seq=0 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f804affd640 1 -- 172.21.15.73:0/2669160686 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c1f15f4080 con 0x55c1f1740970 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f804affd640 10 client.16926 handle_command_reply: tid=0 2024-03-09T00:14:44.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 client.16926 shutdown 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 msgr2=0x55c1f1738a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 secure :-1 s=READY pgs=910 cs=0 l=1 rev1=1 crypto rx=0x7f806409aa70 tx=0x7f806409aba0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 shutdown_connections 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55c1f1740970 0x55c1f1742d90 crc :-1 s=READY pgs=174 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55c1f16d8280 0x55c1f1738a50 unknown :-1 s=CLOSED pgs=910 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55c1f16cf6a0 0x55c1f16decb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 --2- 172.21.15.73:0/2669160686 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55c1f16ced80 0x55c1f16de770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f804affd640 0 client.16926 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.488+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 >> 172.21.15.73:0/2669160686 conn(0x55c1f16c94b0 msgr2=0x55c1f16ca8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 shutdown_connections 2024-03-09T00:14:44.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f807480c740 1 -- 172.21.15.73:0/2669160686 wait complete. 2024-03-09T00:14:44.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f807480c740 20 client.16926 trim_cache size 0 max 16384 2024-03-09T00:14:44.498 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:44.499 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:14:44.499 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "7fbdba11-5a29-446b-bb13-e1ba08d37286", 2024-03-09T00:14:44.499 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:14:44.499 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 msgr2=0x7f80580a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 0x7f80580a4e30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f805c005d90 tx=0x7f805c00a5a0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 msgr2=0x7f806c19ead0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 secure :-1 s=READY pgs=906 cs=0 l=1 rev1=1 crypto rx=0x7f806400b970 tx=0x7f806400be30 comp rx=0 tx=0).stop 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 shutdown_connections 2024-03-09T00:14:44.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f80580a2970 0x7f80580a4e30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f806c19e620 0x7f806c19ead0 unknown :-1 s=CLOSED pgs=906 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f806c1055b0 0x7f806c19a2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 --2- 172.21.15.73:0/3860778007 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f806c100f30 0x7f806c199db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 >> 172.21.15.73:0/3860778007 conn(0x7f806c075530 msgr2=0x7f806c10b8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 shutdown_connections 2024-03-09T00:14:44.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.489+0000 7f8071c30640 1 -- 172.21.15.73:0/3860778007 wait complete. 2024-03-09T00:14:44.502 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:44 smithi073 ceph-mon[32309]: pgmap v243: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:44.502 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:44 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/371607063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:44.503 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:14:44.503 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:14:44.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:44 smithi102 ceph-mon[36106]: pgmap v243: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:44.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:44 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/371607063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:44.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:44 smithi179 ceph-mon[36637]: pgmap v243: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:14:44.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:44 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/371607063' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:14:44.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 -- 172.21.15.73:0/176029842 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc100d80 msgr2=0x7f0cdc104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/176029842 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc100d80 0x7f0cdc104dc0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0009900 tx=0x7f0cd002bc30 comp rx=0 tx=0).stop 2024-03-09T00:14:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 -- 172.21.15.73:0/176029842 shutdown_connections 2024-03-09T00:14:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/176029842 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/176029842 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc100d80 0x7f0cdc104dc0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/176029842 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 -- 172.21.15.73:0/176029842 >> 172.21.15.73:0/176029842 conn(0x7f0cdc0772f0 msgr2=0x7f0cdc077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 -- 172.21.15.73:0/176029842 shutdown_connections 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.614+0000 7f0ce1a68640 1 -- 172.21.15.73:0/176029842 wait complete. 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 Processor -- start 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 -- start start 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc19d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc19dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 0x7f0cdc197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc110530 con 0x7f0cdc1054a0 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1103b0 con 0x7f0cdc1003d0 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0ce1a68640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1106b0 con 0x7f0cdc1979a0 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc19d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc19dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc19d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:56346/0 (socket says 172.21.15.73:56346) 2024-03-09T00:14:44.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/2062882353 learned_addr learned my addr 172.21.15.73:0/2062882353 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.615+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 0x7f0cdc197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 -- 172.21.15.73:0/2062882353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 msgr2=0x7f0cdc197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 0x7f0cdc197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 -- 172.21.15.73:0/2062882353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 msgr2=0x7f0cdc19d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc19d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 -- 172.21.15.73:0/2062882353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc1984c0 con 0x7f0cdc1054a0 2024-03-09T00:14:44.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 0x7f0cdc197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc19dd30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0036040 tx=0x7f0cd002cbf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cd8ff9640 1 -- 172.21.15.73:0/2062882353 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0041020 con 0x7f0cdc1054a0 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cdc198750 con 0x7f0cdc1054a0 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cd8ff9640 1 -- 172.21.15.73:0/2062882353 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cd0004450 con 0x7f0cdc1054a0 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cdc102080 con 0x7f0cdc1054a0 2024-03-09T00:14:44.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.616+0000 7f0cd8ff9640 1 -- 172.21.15.73:0/2062882353 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd0003dd0 con 0x7f0cdc1054a0 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0cd8ff9640 1 -- 172.21.15.73:0/2062882353 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0cd0044020 con 0x7f0cdc1054a0 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0cd8ff9640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 0x7f0cc40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0ce4647740 1 Processor -- start 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 0x7f0cc40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0cd8ff9640 1 -- 172.21.15.73:0/2062882353 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0cd00f0de0 con 0x7f0cdc1054a0 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.619+0000 7f0ce4647740 1 -- start start 2024-03-09T00:14:44.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4f080 0x55d16fc57720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc57c60 0x55d16fc5a050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fbb1e00 con 0x55d16fc4e760 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fb5ab10 con 0x55d16fc57c60 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0ce4647740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fbadea0 con 0x55d16fc4f080 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33642/0 (socket says 172.21.15.73:33642) 2024-03-09T00:14:44.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/2698318532 learned_addr learned my addr 172.21.15.73:0/2698318532 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4f080 0x55d16fc57720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 0x7f0cc40a4e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc198ae0 tx=0x7f0ccc005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc57c60 0x55d16fc5a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/2698318532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4f080 msgr2=0x55d16fc57720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4f080 0x55d16fc57720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/2698318532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc57c60 msgr2=0x55d16fc5a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc57c60 0x55d16fc5a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/2698318532 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d16f9907d0 con 0x55d16fc4e760 2024-03-09T00:14:44.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.620+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009f70 tx=0x7f0ccc014000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8873a0b79cacb30d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.621+0000 7f0cc1ffb640 1 -- 172.21.15.73:0/2698318532 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc0265f0 con 0x55d16fc4e760 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.621+0000 7f0cc1ffb640 1 -- 172.21.15.73:0/2698318532 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc026c70 con 0x55d16fc4e760 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.621+0000 7f0cc1ffb640 1 -- 172.21.15.73:0/2698318532 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc025da0 con 0x55d16fc4e760 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.621+0000 7f0ce4647740 1 -- 172.21.15.73:0/2698318532 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 msgr2=0x55d16fc4eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.621+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 secure :-1 s=READY pgs=913 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc009f70 tx=0x7f0ccc014000 comp rx=0 tx=0).stop 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 -- 172.21.15.73:0/2698318532 shutdown_connections 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc57c60 0x55d16fc5a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4f080 0x55d16fc57720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2698318532 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4e760 0x55d16fc4eb40 unknown :-1 s=CLOSED pgs=913 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 -- 172.21.15.73:0/2698318532 >> 172.21.15.73:0/2698318532 conn(0x55d16fc48e90 msgr2=0x55d16fc49270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 -- 172.21.15.73:0/2698318532 shutdown_connections 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 -- 172.21.15.73:0/2698318532 wait complete. 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 -- start start 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.622+0000 7f0ce4647740 1 Processor -- start 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc54c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fc5aa50 con 0x55d16fc57c60 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fb6e0a0 con 0x55d16fc4f080 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d16fc5abd0 con 0x55d16fc4e760 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0ce4647740 10 client.0 _ll_register_callbacks cb 0x55d16fb6da70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc54c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:56368/0 (socket says 172.21.15.73:56368) 2024-03-09T00:14:44.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdaffd640 1 -- 172.21.15.73:0/2624932118 learned_addr learned my addr 172.21.15.73:0/2624932118 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 -- 172.21.15.73:0/2624932118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 msgr2=0x55d16fc573a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc573a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 -- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 msgr2=0x55d16fc54730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdbfff640 1 -- 172.21.15.73:0/2624932118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d16f8c63f0 con 0x55d16fc57c60 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.623+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:44.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc54c70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4005fd0 tx=0x7f0cd4004410 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f5a7bd15d8460df server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd4023030 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ccc026c70 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0cd4004b20 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cd402d050 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 10 client.16938 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d16f8860f0 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ccc026c70 con 0x55d16fc57c60 2024-03-09T00:14:44.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0cd4028840 con 0x55d16fc57c60 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 10 client.16938 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 10 client.16938 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 20 client.16938 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fc67750 0x55d16fc69b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 4 client.16938 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.624+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d16fbbf340 con 0x55d16fc67750 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.625+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fc67750 0x55d16fc69b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.625+0000 7f0cbaffd640 10 client.16938 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.625+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fc67750 0x55d16fc69b40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e77769df83b5b8ab server_cookie=999ad8ac23a55078 in_seq=0 out_seq=0 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0cbaffd640 1 -- 172.21.15.73:0/2624932118 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55d16fbbf340 con 0x55d16fc67750 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0cbaffd640 10 client.16938 handle_command_reply: tid=0 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 client.16938 shutdown 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 msgr2=0x55d16fc54c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc54c70 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 crypto rx=0x7f0cd4005fd0 tx=0x7f0cd4004410 comp rx=0 tx=0).stop 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 shutdown_connections 2024-03-09T00:14:44.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fc67750 0x55d16fc69b40 crc :-1 s=READY pgs=175 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc54c70 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc54730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0cbaffd640 0 client.16938 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 --2- 172.21.15.73:0/2624932118 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc573a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 >> 172.21.15.73:0/2624932118 conn(0x55d16fc48e90 msgr2=0x55d16fc4abe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 shutdown_connections 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.626+0000 7f0ce4647740 1 -- 172.21.15.73:0/2624932118 wait complete. 2024-03-09T00:14:44.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.627+0000 7f0ce4647740 20 client.16938 trim_cache size 0 max 16384 2024-03-09T00:14:44.642 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.643+0000 7f0ce4647740 1 Processor -- start 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.643+0000 7f0ce4647740 1 -- start start 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.643+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.643+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc573a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.643+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0ce4647740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1aa640 con 0x55d16fc57c60 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0ce4647740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1acbe0 con 0x55d16fc4f080 2024-03-09T00:14:44.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0ce4647740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1aac40 con 0x55d16fc4e760 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47860/0 (socket says 172.21.15.73:47860) 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc573a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/1062285033 learned_addr learned my addr 172.21.15.73:0/1062285033 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdbfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33654/0 (socket says 172.21.15.73:33654) 2024-03-09T00:14:44.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 -- 172.21.15.73:0/1062285033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 msgr2=0x55d16fc56e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 -- 172.21.15.73:0/1062285033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 msgr2=0x55d16fc5ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 -- 172.21.15.73:0/1062285033 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d16fc4eeb0 con 0x55d16fc4f080 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:44.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc573a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005820 tx=0x7f0cd00f0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1956b55cb1df5f87 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.644+0000 7f0cc17fa640 1 -- 172.21.15.73:0/1062285033 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00bcac0 con 0x55d16fc4f080 2024-03-09T00:14:44.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0cc17fa640 1 -- 172.21.15.73:0/1062285033 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cd00bd860 con 0x55d16fc4f080 2024-03-09T00:14:44.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0cc17fa640 1 -- 172.21.15.73:0/1062285033 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cd00b97d0 con 0x55d16fc4f080 2024-03-09T00:14:44.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 -- 172.21.15.73:0/1062285033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 msgr2=0x55d16fc573a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc573a0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0005820 tx=0x7f0cd00f0040 comp rx=0 tx=0).stop 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 -- 172.21.15.73:0/1062285033 shutdown_connections 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc57c60 0x55d16fc5ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4f080 0x55d16fc573a0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 --2- 172.21.15.73:0/1062285033 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc4e760 0x55d16fc56e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 -- 172.21.15.73:0/1062285033 >> 172.21.15.73:0/1062285033 conn(0x55d16fc48e90 msgr2=0x55d16fc546f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 -- 172.21.15.73:0/1062285033 shutdown_connections 2024-03-09T00:14:44.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.645+0000 7f0ce4647740 1 -- 172.21.15.73:0/1062285033 wait complete. 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 -- start start 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 Processor -- start 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 0x55d16fc5eef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc57c60 0x55d16fc5b920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1ab260 con 0x55d16fc4f080 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1ac580 con 0x55d16fc4e760 2024-03-09T00:14:44.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cdc1ab8c0 con 0x55d16fc57c60 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0ce4647740 10 client.0 _ll_register_callbacks cb 0x7f0cd4003a50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0cdb7fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 0x55d16fc5eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0cdaffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:33656/0 (socket says 172.21.15.73:33656) 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0cdaffd640 1 -- 172.21.15.73:0/143609287 learned_addr learned my addr 172.21.15.73:0/143609287 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.646+0000 7f0cdbfff640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc57c60 0x55d16fc5b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/143609287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc57c60 msgr2=0x55d16fc5b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc57c60 0x55d16fc5b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/143609287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 msgr2=0x55d16fc5b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 -- 172.21.15.73:0/143609287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d16fc5be60 con 0x55d16fc4e760 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cdb7fe640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 0x55d16fc5eef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc01e010 tx=0x7f0ccc005d90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d396af4545674fe server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cbaffd640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ccc03c070 con 0x55d16fc4e760 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0cd00bd860 con 0x55d16fc4e760 2024-03-09T00:14:44.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0ccc025470 con 0x55d16fc4e760 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ccc036e60 con 0x55d16fc4e760 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0ce4647740 10 client.26135 fetch_fsmap learned FSMap version 20 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55d16fcbe980 con 0x55d16fc4e760 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.647+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cd00bd860 con 0x55d16fc4e760 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f0ccc036190 con 0x55d16fc4e760 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 10 client.26135 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 10 client.26135 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 20 client.26135 populate_metadata read hostname 'smithi073' 2024-03-09T00:14:44.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fcbed20 0x55d16fcc1140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 4 client.26135 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55d16fbb9690 con 0x55d16fcbed20 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fcbed20 0x55d16fcc1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0cbaffd640 10 client.26135 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.648+0000 7f0cdaffd640 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fcbed20 0x55d16fcc1140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7ff17a738db5d70 server_cookie=d7b87a7c42048b41 in_seq=0 out_seq=0 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0cbaffd640 1 -- 172.21.15.73:0/143609287 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55d16fbb9690 con 0x55d16fcbed20 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0cbaffd640 10 client.26135 handle_command_reply: tid=0 2024-03-09T00:14:44.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 client.26135 shutdown 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 msgr2=0x55d16fc5eef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 0x55d16fc5eef0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f0ccc01e010 tx=0x7f0ccc005d90 comp rx=0 tx=0).stop 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 shutdown_connections 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55d16fcbed20 0x55d16fcc1140 crc :-1 s=READY pgs=176 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55d16fc57c60 0x55d16fc5b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0cbaffd640 0 client.26135 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55d16fc4f080 0x55d16fc5b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 --2- 172.21.15.73:0/143609287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55d16fc4e760 0x55d16fc5eef0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 >> 172.21.15.73:0/143609287 conn(0x55d16fc48e90 msgr2=0x55d16fc52ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 shutdown_connections 2024-03-09T00:14:44.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 1 -- 172.21.15.73:0/143609287 wait complete. 2024-03-09T00:14:44.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.649+0000 7f0ce4647740 20 client.26135 trim_cache size 0 max 16384 2024-03-09T00:14:44.658 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2388 inodes in the stack)", 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "7fbdba11-5a29-446b-bb13-e1ba08d37286": { 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "7fbdba11-5a29-446b-bb13-e1ba08d37286", 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:14:44.659 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 msgr2=0x7f0cc40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 0x7f0cc40a4e30 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f0cdc198ae0 tx=0x7f0ccc005f90 comp rx=0 tx=0).stop 2024-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 msgr2=0x7f0cdc19dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:14:44.661 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc19dd30 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 crypto rx=0x7f0cd0036040 tx=0x7f0cd002cbf0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 shutdown_connections 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0cdc1979a0 0x7f0cdc197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f0cc40a2970 0x7f0cc40a4e30 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0cdc1054a0 0x7f0cdc19dd30 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 --2- 172.21.15.73:0/2062882353 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0cdc1003d0 0x7f0cdc19d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 >> 172.21.15.73:0/2062882353 conn(0x7f0cdc0772f0 msgr2=0x7f0cdc0687f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:14:44.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 shutdown_connections 2024-03-09T00:14:44.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:14:44.650+0000 7f0ce1a68640 1 -- 172.21.15.73:0/2062882353 wait complete. 2024-03-09T00:14:44.663 INFO:tasks.cephfs.filesystem:scrub status for tag:7fbdba11-5a29-446b-bb13-e1ba08d37286 - {'path': '/', 'tag': '7fbdba11-5a29-446b-bb13-e1ba08d37286', 'options': 'recursive,force'} 2024-03-09T00:14:45.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:45 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:14:45.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:45 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:45 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:14:45.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:45 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:45 smithi102 ceph-mon[36106]: pgmap v244: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:45.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:45 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:14:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:45 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:45 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:14:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:45 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:45 smithi179 ceph-mon[36637]: pgmap v244: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:45.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:45 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:14:45.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:45 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:45 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:14:45.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:45 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:14:45.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:45 smithi073 ceph-mon[32309]: pgmap v244: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:14:48.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:48 smithi073 ceph-mon[32309]: pgmap v245: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 182 op/s 2024-03-09T00:14:48.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:48 smithi102 ceph-mon[36106]: pgmap v245: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 182 op/s 2024-03-09T00:14:48.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:48 smithi179 ceph-mon[36637]: pgmap v245: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 181 KiB/s rd, 182 op/s 2024-03-09T00:14:49.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:49 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:14:49.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:49 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:14:49.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:49 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:14:50.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:50 smithi102 ceph-mon[36106]: pgmap v246: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:50.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:50 smithi179 ceph-mon[36637]: pgmap v246: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:50.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:50 smithi073 ceph-mon[32309]: pgmap v246: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:52.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:52 smithi102 ceph-mon[36106]: pgmap v247: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:52.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:52 smithi179 ceph-mon[36637]: pgmap v247: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:52.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:52 smithi073 ceph-mon[32309]: pgmap v247: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:54.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:54 smithi102 ceph-mon[36106]: pgmap v248: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:54.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:54 smithi179 ceph-mon[36637]: pgmap v248: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:54.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:54 smithi073 ceph-mon[32309]: pgmap v248: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:56.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:56 smithi102 ceph-mon[36106]: pgmap v249: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:56.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:56 smithi179 ceph-mon[36637]: pgmap v249: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:56.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:56 smithi073 ceph-mon[32309]: pgmap v249: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:14:58.456 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:58 smithi073 ceph-mon[32309]: pgmap v250: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:58 smithi102 ceph-mon[36106]: pgmap v250: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:58.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:58 smithi179 ceph-mon[36637]: pgmap v250: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:14:59.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:14:59 smithi102 ceph-mon[36106]: pgmap v251: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:14:59.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:14:59 smithi179 ceph-mon[36637]: pgmap v251: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:14:59.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:14:59 smithi073 ceph-mon[32309]: pgmap v251: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:15:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:02 smithi102 ceph-mon[36106]: pgmap v252: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:02.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:15:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:02 smithi179 ceph-mon[36637]: pgmap v252: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:02.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:15:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:02 smithi073 ceph-mon[32309]: pgmap v252: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:02.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:15:03.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:15:03.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:15:03.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:15:03.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:15:03.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:15:03.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:15:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:15:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:15:03.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:15:04.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:04 smithi102 ceph-mon[36106]: pgmap v253: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:04.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:04 smithi179 ceph-mon[36637]: pgmap v253: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:04.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:04 smithi073 ceph-mon[32309]: pgmap v253: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:06.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:06 smithi102 ceph-mon[36106]: pgmap v254: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:06.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:06 smithi179 ceph-mon[36637]: pgmap v254: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:06.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:06 smithi073 ceph-mon[32309]: pgmap v254: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:08.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:08 smithi102 ceph-mon[36106]: pgmap v255: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:08.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:08 smithi179 ceph-mon[36637]: pgmap v255: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:08.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:08 smithi073 ceph-mon[32309]: pgmap v255: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:15:09.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:09 smithi102 ceph-mon[36106]: pgmap v256: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:09.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:09 smithi102 ceph-mon[36106]: osdmap e99: 12 total, 12 up, 12 in 2024-03-09T00:15:09.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:09 smithi179 ceph-mon[36637]: pgmap v256: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:09.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:09 smithi179 ceph-mon[36637]: osdmap e99: 12 total, 12 up, 12 in 2024-03-09T00:15:09.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:09 smithi073 ceph-mon[32309]: pgmap v256: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:15:09.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:09 smithi073 ceph-mon[32309]: osdmap e99: 12 total, 12 up, 12 in 2024-03-09T00:15:12.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:12 smithi102 ceph-mon[36106]: pgmap v258: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:15:12.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:12 smithi179 ceph-mon[36637]: pgmap v258: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:15:12.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:12 smithi073 ceph-mon[32309]: pgmap v258: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 14 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:15:14.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:14 smithi102 ceph-mon[36106]: pgmap v259: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:15:14.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:14 smithi179 ceph-mon[36637]: pgmap v259: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:15:14.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:14 smithi073 ceph-mon[32309]: pgmap v259: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:15:14.664 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:15:14.665 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:15:14.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 -- 172.21.15.73:0/538132616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1013d0 msgr2=0x7f11cc1017b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/538132616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1013d0 0x7f11cc1017b0 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 crypto rx=0x7f11bc009900 tx=0x7f11bc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:14.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 -- 172.21.15.73:0/538132616 shutdown_connections 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/538132616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc1064a0 0x7f11cc10e070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/538132616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc101d80 0x7f11cc105dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/538132616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1013d0 0x7f11cc1017b0 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.782+0000 7f11d0d9f640 1 -- 172.21.15.73:0/538132616 >> 172.21.15.73:0/538132616 conn(0x7f11cc0fcc50 msgr2=0x7f11cc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.783+0000 7f11d0d9f640 1 -- 172.21.15.73:0/538132616 shutdown_connections 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.783+0000 7f11d0d9f640 1 -- 172.21.15.73:0/538132616 wait complete. 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.783+0000 7f11d0d9f640 1 Processor -- start 2024-03-09T00:15:14.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.783+0000 7f11d0d9f640 1 -- start start 2024-03-09T00:15:14.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 0x7f11cc19d6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc197870 0x7f11cc197d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc111550 con 0x7f11cc1064a0 2024-03-09T00:15:14.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1113d0 con 0x7f11cc197870 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1116d0 con 0x7f11cc1013d0 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52630/0 (socket says 172.21.15.73:52630) 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 -- 172.21.15.73:0/301255176 learned_addr learned my addr 172.21.15.73:0/301255176 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11cbfff640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc197870 0x7f11cc197d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 0x7f11cc19d6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 -- 172.21.15.73:0/301255176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 msgr2=0x7f11cc19d6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 0x7f11cc19d6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 -- 172.21.15.73:0/301255176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc197870 msgr2=0x7f11cc197d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc197870 0x7f11cc197d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 -- 172.21.15.73:0/301255176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11cc198390 con 0x7f11cc1064a0 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 0x7f11cc19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f11b400d740 tx=0x7f11b400dc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c97fa640 1 -- 172.21.15.73:0/301255176 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4017070 con 0x7f11cc1064a0 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11cc198680 con 0x7f11cc1064a0 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c97fa640 1 -- 172.21.15.73:0/301255176 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11b400de50 con 0x7f11cc1064a0 2024-03-09T00:15:14.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.784+0000 7f11c97fa640 1 -- 172.21.15.73:0/301255176 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4004e00 con 0x7f11cc1064a0 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.785+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11cc1a48d0 con 0x7f11cc1064a0 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11c97fa640 1 -- 172.21.15.73:0/301255176 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f11b4004fa0 con 0x7f11cc1064a0 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 Processor -- start 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 -- start start 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11c97fa640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 0x7f11b80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af248c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af271f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14ae7ee00 con 0x55f14af1b900 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14ae27b10 con 0x55f14af24e00 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14ae7aea0 con 0x55f14af1c220 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11cb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11cbfff640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 0x7f11b80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11cb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52634/0 (socket says 172.21.15.73:52634) 2024-03-09T00:15:14.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11cb7fe640 1 -- 172.21.15.73:0/3340224616 learned_addr learned my addr 172.21.15.73:0/3340224616 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11c97fa640 1 -- 172.21.15.73:0/301255176 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f11b40cec50 con 0x7f11cc1064a0 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.787+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af248c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cbfff640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af271f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 -- 172.21.15.73:0/3340224616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 msgr2=0x55f14af248c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af248c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 -- 172.21.15.73:0/3340224616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 msgr2=0x55f14af271f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af271f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 -- 172.21.15.73:0/3340224616 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14ac5d7d0 con 0x55f14af1b900 2024-03-09T00:15:14.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cbfff640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f11bc000c00 tx=0x7f11bc02c720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d4413f6ebe5dc3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11cbfff640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 0x7f11b80a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f14af171c0 tx=0x7f11c4007940 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11c1ffb640 1 -- 172.21.15.73:0/3340224616 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc02cbc0 con 0x55f14af1b900 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11c1ffb640 1 -- 172.21.15.73:0/3340224616 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11bc005040 con 0x55f14af1b900 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.788+0000 7f11c1ffb640 1 -- 172.21.15.73:0/3340224616 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc004970 con 0x55f14af1b900 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3340224616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 msgr2=0x55f14af1bce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 crypto rx=0x7f11bc000c00 tx=0x7f11bc02c720 comp rx=0 tx=0).stop 2024-03-09T00:15:14.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3340224616 shutdown_connections 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af271f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af248c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3340224616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af1bce0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3340224616 >> 172.21.15.73:0/3340224616 conn(0x55f14af16030 msgr2=0x55f14af16410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3340224616 shutdown_connections 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3340224616 wait complete. 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 -- start start 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 Processor -- start 2024-03-09T00:15:14.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.789+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af21d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14af27b90 con 0x55f14af1c220 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14ae3b0a0 con 0x55f14af1b900 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f14af27d10 con 0x55f14af24e00 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 10 client.0 _ll_register_callbacks cb 0x55f14ae3aa70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52638/0 (socket says 172.21.15.73:52638) 2024-03-09T00:15:14.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 -- 172.21.15.73:0/3527948863 learned_addr learned my addr 172.21.15.73:0/3527948863 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11cbfff640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af21d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 -- 172.21.15.73:0/3527948863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 msgr2=0x55f14af21d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af21d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 -- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 msgr2=0x55f14af244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af244c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 -- 172.21.15.73:0/3527948863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f14ab933f0 con 0x55f14af1c220 2024-03-09T00:15:14.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11cbfff640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af21d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f11b40972d0 tx=0x7f11b4004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7bfff3c651bcd251 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4020040 con 0x55f14af1c220 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f11b40974e0 con 0x55f14af1c220 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f11bc005040 con 0x55f14af1c220 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11b409efc0 con 0x55f14af1c220 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 10 client.16956 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.790+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f14ab530f0 con 0x55f14af1c220 2024-03-09T00:15:14.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11bc005040 con 0x55f14af1c220 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11b406b8b0 con 0x55f14af1c220 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 10 client.16956 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 10 client.16956 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 20 client.16956 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af34740 0x55f14af36b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 4 client.16956 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:14.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f14ae8c340 con 0x55f14af34740 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.791+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af34740 0x55f14af36b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11aaffd640 10 client.16956 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af34740 0x55f14af36b60 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2f7ea1a987552b83 server_cookie=b6f81336d58d7bc5 in_seq=0 out_seq=0 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11aaffd640 1 -- 172.21.15.73:0/3527948863 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55f14ae8c340 con 0x55f14af34740 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11aaffd640 10 client.16956 handle_command_reply: tid=0 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 client.16956 shutdown 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 msgr2=0x55f14af21850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 secure :-1 s=READY pgs=918 cs=0 l=1 rev1=1 crypto rx=0x7f11b40972d0 tx=0x7f11b4004800 comp rx=0 tx=0).stop 2024-03-09T00:15:14.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 shutdown_connections 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af34740 0x55f14af36b60 crc :-1 s=READY pgs=177 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af21d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af21850 unknown :-1 s=CLOSED pgs=918 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/3527948863 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.792+0000 7f11aaffd640 0 client.16956 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.793+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 >> 172.21.15.73:0/3527948863 conn(0x55f14af16030 msgr2=0x55f14af17d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.793+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 shutdown_connections 2024-03-09T00:15:14.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.793+0000 7f11d4bdb740 1 -- 172.21.15.73:0/3527948863 wait complete. 2024-03-09T00:15:14.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.793+0000 7f11d4bdb740 20 client.16956 trim_cache size 0 max 16384 2024-03-09T00:15:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 Processor -- start 2024-03-09T00:15:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 -- start start 2024-03-09T00:15:14.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af20830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af244c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1ac520 con 0x55f14af1c220 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1aeac0 con 0x55f14af1b900 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.807+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1acb20 con 0x55f14af24e00 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52648/0 (socket says 172.21.15.73:52648) 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 -- 172.21.15.73:0/1629216209 learned_addr learned my addr 172.21.15.73:0/1629216209 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af20830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 -- 172.21.15.73:0/1629216209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 msgr2=0x55f14af244c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af244c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 -- 172.21.15.73:0/1629216209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 msgr2=0x55f14af20830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af20830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 -- 172.21.15.73:0/1629216209 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f14af17260 con 0x55f14af1c220 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af20830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f11b4005d00 tx=0x7f11b40048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7480cbce958754f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c17fa640 1 -- 172.21.15.73:0/1629216209 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b4020040 con 0x55f14af1c220 2024-03-09T00:15:14.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c17fa640 1 -- 172.21.15.73:0/1629216209 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11b409a2c0 con 0x55f14af1c220 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.808+0000 7f11c17fa640 1 -- 172.21.15.73:0/1629216209 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11b406ad30 con 0x55f14af1c220 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- 172.21.15.73:0/1629216209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 msgr2=0x55f14af23f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 crypto rx=0x7f11b4005d00 tx=0x7f11b40048a0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- 172.21.15.73:0/1629216209 shutdown_connections 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af24e00 0x55f14af244c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1c220 0x55f14af23f80 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/1629216209 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af1b900 0x55f14af20830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- 172.21.15.73:0/1629216209 >> 172.21.15.73:0/1629216209 conn(0x55f14af16030 msgr2=0x55f14af20120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- 172.21.15.73:0/1629216209 shutdown_connections 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- 172.21.15.73:0/1629216209 wait complete. 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- start start 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 Processor -- start 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af84550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af291e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1ad140 con 0x55f14af1b900 2024-03-09T00:15:14.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1ae460 con 0x55f14af24e00 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11cc1ad7a0 con 0x55f14af1c220 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.809+0000 7f11d4bdb740 10 client.0 _ll_register_callbacks cb 0x7f11b409a7d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52652/0 (socket says 172.21.15.73:52652) 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 -- 172.21.15.73:0/2705828608 learned_addr learned my addr 172.21.15.73:0/2705828608 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af84550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 -- 172.21.15.73:0/2705828608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 msgr2=0x55f14af84550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af84550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 -- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 msgr2=0x55f14af291e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af291e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 -- 172.21.15.73:0/2705828608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f14af299c0 con 0x55f14af1b900 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af84550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11cb7fe640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f11bc006010 tx=0x7f11bc02ce00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2bbe06cee26a951 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11aaffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f11bc04f650 con 0x55f14af1b900 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9613+0+0 (secure 0 0 0) 0x7f11bc04f7f0 con 0x55f14af1b900 2024-03-09T00:15:14.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55f14ae86690 con 0x55f14af1b900 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f11bc05d050 con 0x55f14af1b900 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11d4bdb740 10 client.16962 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55f14af8c740 con 0x55f14af1b900 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.810+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55f14ae86690 con 0x55f14af1b900 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f11bc0046c0 con 0x55f14af1b900 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 10 client.16962 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 10 client.16962 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:14.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 20 client.16962 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af8cad0 0x55f14af8eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 4 client.16962 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f14ae044e0 con 0x55f14af8cad0 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af8cad0 0x55f14af8eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11aaffd640 10 client.16962 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11c2ffd640 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af8cad0 0x55f14af8eec0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b48bf676cf96e9c9 server_cookie=6ea18d2c25f20c71 in_seq=0 out_seq=0 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11aaffd640 1 -- 172.21.15.73:0/2705828608 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55f14ae044e0 con 0x55f14af8cad0 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11aaffd640 10 client.16962 handle_command_reply: tid=0 2024-03-09T00:15:14.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.811+0000 7f11d4bdb740 1 client.16962 shutdown 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 msgr2=0x55f14af84010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 secure :-1 s=READY pgs=920 cs=0 l=1 rev1=1 crypto rx=0x7f11bc006010 tx=0x7f11bc02ce00 comp rx=0 tx=0).stop 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 shutdown_connections 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55f14af8cad0 0x55f14af8eec0 crc :-1 s=READY pgs=178 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55f14af24e00 0x55f14af291e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55f14af1c220 0x55f14af84550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11aaffd640 0 client.16962 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 --2- 172.21.15.73:0/2705828608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55f14af1b900 0x55f14af84010 unknown :-1 s=CLOSED pgs=920 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 >> 172.21.15.73:0/2705828608 conn(0x55f14af16030 msgr2=0x55f14af209b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 shutdown_connections 2024-03-09T00:15:14.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 1 -- 172.21.15.73:0/2705828608 wait complete. 2024-03-09T00:15:14.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d4bdb740 20 client.16962 trim_cache size 0 max 16384 2024-03-09T00:15:14.821 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:14.822 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:15:14.822 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:15:14.822 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 msgr2=0x7f11b80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 0x7f11b80a4e30 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x55f14af171c0 tx=0x7f11c4007940 comp rx=0 tx=0).stop 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 msgr2=0x7f11cc19dc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 secure :-1 s=READY pgs=916 cs=0 l=1 rev1=1 crypto rx=0x7f11b400d740 tx=0x7f11b400dc00 comp rx=0 tx=0).stop 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 shutdown_connections 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f11cc197870 0x7f11cc197d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f11b80a2970 0x7f11b80a4e30 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f11cc1064a0 0x7f11cc19dc00 unknown :-1 s=CLOSED pgs=916 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 --2- 172.21.15.73:0/301255176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f11cc1013d0 0x7f11cc19d6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 >> 172.21.15.73:0/301255176 conn(0x7f11cc0fcc50 msgr2=0x7f11cc0fef80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 shutdown_connections 2024-03-09T00:15:14.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.812+0000 7f11d0d9f640 1 -- 172.21.15.73:0/301255176 wait complete. 2024-03-09T00:15:14.825 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:15:14.826 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:15:14.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.928+0000 7fea28427640 1 -- 172.21.15.73:0/589353854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea2010b930 msgr2=0x7fea2010ddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.928+0000 7fea28427640 1 --2- 172.21.15.73:0/589353854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea2010b930 0x7fea2010ddd0 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 crypto rx=0x7fea1c009900 tx=0x7fea1c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:14.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- 172.21.15.73:0/589353854 shutdown_connections 2024-03-09T00:15:14.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- 172.21.15.73:0/589353854 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea2010b930 0x7fea2010ddd0 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- 172.21.15.73:0/589353854 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea20102e80 0x7fea2010b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- 172.21.15.73:0/589353854 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 0x7fea20102940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- 172.21.15.73:0/589353854 >> 172.21.15.73:0/589353854 conn(0x7fea200772f0 msgr2=0x7fea20077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- 172.21.15.73:0/589353854 shutdown_connections 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- 172.21.15.73:0/589353854 wait complete. 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 Processor -- start 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- start start 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 0x7fea2019d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 0x7fea20197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20111710 con 0x7fea20102e80 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20111590 con 0x7fea2010b930 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea28427640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea20111890 con 0x7fea20102560 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea259c2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea259c2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52670/0 (socket says 172.21.15.73:52670) 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.929+0000 7fea259c2640 1 -- 172.21.15.73:0/4113903046 learned_addr learned my addr 172.21.15.73:0/4113903046 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:14.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea269c4640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 0x7fea20197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea261c3640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 0x7fea2019d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 -- 172.21.15.73:0/4113903046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 msgr2=0x7fea2019d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 0x7fea2019d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 -- 172.21.15.73:0/4113903046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 msgr2=0x7fea20197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 0x7fea20197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 -- 172.21.15.73:0/4113903046 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea20198110 con 0x7fea20102e80 2024-03-09T00:15:14.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea269c4640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 0x7fea20197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea259c2640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fea1000e9f0 tx=0x7fea1000eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea1000cd80 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea10010910 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea20198400 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea10018990 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.930+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea201a46a0 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.931+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea200680e0 con 0x7fea20102e80 2024-03-09T00:15:14.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.932+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fea10010ac0 con 0x7fea20102e80 2024-03-09T00:15:14.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.933+0000 7fea177fe640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 0x7fea0c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:14.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.933+0000 7fea261c3640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 0x7fea0c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:14.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.933+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fea100cfb20 con 0x7fea20102e80 2024-03-09T00:15:14.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.933+0000 7fea261c3640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 0x7fea0c0a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fea20198ad0 tx=0x7fea180094a0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:14.939 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:14.937+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fea1009c440 con 0x7fea20102e80 2024-03-09T00:15:15.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.153+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fea201a48d0 con 0x7fea20102e80 2024-03-09T00:15:15.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.153+0000 7fea177fe640 1 -- 172.21.15.73:0/4113903046 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fea1009bd40 con 0x7fea20102e80 2024-03-09T00:15:15.152 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:15:15.152 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:15:15.152 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 msgr2=0x7fea0c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 0x7fea0c0a4e30 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7fea20198ad0 tx=0x7fea180094a0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 msgr2=0x7fea2019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 crypto rx=0x7fea1000e9f0 tx=0x7fea1000eeb0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 shutdown_connections 2024-03-09T00:15:15.154 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fea2010b930 0x7fea20197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fea0c0a2970 0x7fea0c0a4e30 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fea20102e80 0x7fea2019dd20 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 --2- 172.21.15.73:0/4113903046 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fea20102560 0x7fea2019d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.155+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 >> 172.21.15.73:0/4113903046 conn(0x7fea200772f0 msgr2=0x7fea200fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.156+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 shutdown_connections 2024-03-09T00:15:15.155 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.156+0000 7fea28427640 1 -- 172.21.15.73:0/4113903046 wait complete. 2024-03-09T00:15:15.164 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:15:15.165 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:15:15.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 -- 172.21.15.73:0/1412265378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 msgr2=0x7fb7dc10c810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 --2- 172.21.15.73:0/1412265378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc10c810 secure :-1 s=READY pgs=923 cs=0 l=1 rev1=1 crypto rx=0x7fb7d400ade0 tx=0x7fb7d402cfd0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 -- 172.21.15.73:0/1412265378 shutdown_connections 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 --2- 172.21.15.73:0/1412265378 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc10c810 unknown :-1 s=CLOSED pgs=923 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 --2- 172.21.15.73:0/1412265378 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc0ff940 0x7fb7dc0ffdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 --2- 172.21.15.73:0/1412265378 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc105950 0x7fb7dc105d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 -- 172.21.15.73:0/1412265378 >> 172.21.15.73:0/1412265378 conn(0x7fb7dc076710 msgr2=0x7fb7dc0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 -- 172.21.15.73:0/1412265378 shutdown_connections 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.274+0000 7fb7e4085640 1 -- 172.21.15.73:0/1412265378 wait complete. 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 Processor -- start 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 -- start start 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc0ff940 0x7fb7dc19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 0x7fb7dc197950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc10f4e0 con 0x7fb7dc100300 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc10f360 con 0x7fb7dc105950 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e4085640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc10f660 con 0x7fb7dc0ff940 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52688/0 (socket says 172.21.15.73:52688) 2024-03-09T00:15:15.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 -- 172.21.15.73:0/3033378999 learned_addr learned my addr 172.21.15.73:0/3033378999 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc0ff940 0x7fb7dc19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 0x7fb7dc197950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 -- 172.21.15.73:0/3033378999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc0ff940 msgr2=0x7fb7dc19d7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc0ff940 0x7fb7dc19d7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 -- 172.21.15.73:0/3033378999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 msgr2=0x7fb7dc197950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 0x7fb7dc197950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 -- 172.21.15.73:0/3033378999 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7dc1980d0 con 0x7fb7dc100300 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 0x7fb7dc197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00d950 tx=0x7fb7cc00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.275+0000 7fb7d2ffd640 1 -- 172.21.15.73:0/3033378999 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc017030 con 0x7fb7dc100300 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.276+0000 7fb7d2ffd640 1 -- 172.21.15.73:0/3033378999 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7cc004020 con 0x7fb7dc100300 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.276+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7dc1983c0 con 0x7fb7dc100300 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.276+0000 7fb7d2ffd640 1 -- 172.21.15.73:0/3033378999 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7cc0047a0 con 0x7fb7dc100300 2024-03-09T00:15:15.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.276+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7dc1a4660 con 0x7fb7dc100300 2024-03-09T00:15:15.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7d2ffd640 1 -- 172.21.15.73:0/3033378999 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb7cc0041d0 con 0x7fb7dc100300 2024-03-09T00:15:15.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 Processor -- start 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 -- start start 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7d2ffd640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 0x7fb7c80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b73160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b736a0 0x5556c0b75a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0b31360 con 0x5556c0b6a1a0 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0acdec0 con 0x5556c0b736a0 2024-03-09T00:15:15.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.278+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0a76d30 con 0x5556c0b6aac0 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 0x7fb7c80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7d2ffd640 1 -- 172.21.15.73:0/3033378999 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb7cc0cec40 con 0x7fb7dc100300 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52692/0 (socket says 172.21.15.73:52692) 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 -- 172.21.15.73:0/2463142631 learned_addr learned my addr 172.21.15.73:0/2463142631 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1620640 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b73160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 -- 172.21.15.73:0/2463142631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 msgr2=0x5556c0b73160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b73160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 -- 172.21.15.73:0/2463142631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b736a0 msgr2=0x5556c0b75a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b736a0 0x5556c0b75a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 -- 172.21.15.73:0/2463142631 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556c0a75bc0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1620640 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b73160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 0x7fb7c80a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4038040 tx=0x7fb7d4033340 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8004650 tx=0x7fb7d800bb20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c7aee2b836950e80 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7c7fff640 1 -- 172.21.15.73:0/2463142631 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8003bd0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7c7fff640 1 -- 172.21.15.73:0/2463142631 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d8015070 con 0x5556c0b6a1a0 2024-03-09T00:15:15.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.279+0000 7fb7c7fff640 1 -- 172.21.15.73:0/2463142631 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d8004db0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2463142631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 msgr2=0x5556c0b6a580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 crypto rx=0x7fb7d8004650 tx=0x7fb7d800bb20 comp rx=0 tx=0).stop 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2463142631 shutdown_connections 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b736a0 0x5556c0b75a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b73160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2463142631 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0b6a580 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2463142631 >> 172.21.15.73:0/2463142631 conn(0x5556c0b648d0 msgr2=0x5556c0b64cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2463142631 shutdown_connections 2024-03-09T00:15:15.290 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2463142631 wait complete. 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 -- start start 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.280+0000 7fb7e6c6e740 1 Processor -- start 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 0x5556c0b70690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0b70bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0a6c4a0 con 0x5556c0b736a0 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0ac9f60 con 0x5556c0b6aac0 2024-03-09T00:15:15.291 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5556c0b76490 con 0x5556c0b6a1a0 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e6c6e740 10 client.0 _ll_register_callbacks cb 0x5556c0ac7d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52698/0 (socket says 172.21.15.73:52698) 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 -- 172.21.15.73:0/3764589552 learned_addr learned my addr 172.21.15.73:0/3764589552 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 0x5556c0b70690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0b70bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 -- 172.21.15.73:0/3764589552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 msgr2=0x5556c0b70690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 0x5556c0b70690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.292 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 -- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 msgr2=0x5556c0b70bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0b70bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 -- 172.21.15.73:0/3764589552 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556c07e23f0 con 0x5556c0b736a0 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 0x5556c0b70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0b70bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7e2622640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4037000 tx=0x7fb7d402cfd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=de15f0d16c96455a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4003e40 con 0x5556c0b736a0 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.281+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb7d4004040 con 0x5556c0b736a0 2024-03-09T00:15:15.293 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7d8015070 con 0x5556c0b736a0 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d4004240 con 0x5556c0b736a0 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 10 client.16980 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5556c08ac7d0 con 0x5556c0b736a0 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d8015070 con 0x5556c0b736a0 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb7d404b070 con 0x5556c0b736a0 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 10 client.16980 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 10 client.16980 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:15.294 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 20 client.16980 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0b83790 0x5556c0b85b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 4 client.16980 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5556c0adb400 con 0x5556c0b83790 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.282+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0b83790 0x5556c0b85b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.283+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0b83790 0x5556c0b85b80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4f91e1ef26c9f44d server_cookie=4cd3a91684b677d in_seq=0 out_seq=0 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.283+0000 7fb7c57fa640 10 client.16980 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.283+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3764589552 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5556c0adb400 con 0x5556c0b83790 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.283+0000 7fb7c57fa640 10 client.16980 handle_command_reply: tid=0 2024-03-09T00:15:15.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.283+0000 7fb7e6c6e740 1 client.16980 shutdown 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 msgr2=0x5556c0b71110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 secure :-1 s=READY pgs=926 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4037000 tx=0x7fb7d402cfd0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 shutdown_connections 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0b83790 0x5556c0b85b80 crc :-1 s=READY pgs=179 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b71110 unknown :-1 s=CLOSED pgs=926 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0b70bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7c57fa640 0 client.16980 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:15.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3764589552 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6a1a0 0x5556c0b70690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 >> 172.21.15.73:0/3764589552 conn(0x5556c0b648d0 msgr2=0x5556c0b66620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 shutdown_connections 2024-03-09T00:15:15.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3764589552 wait complete. 2024-03-09T00:15:15.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.284+0000 7fb7e6c6e740 20 client.16980 trim_cache size 0 max 16384 2024-03-09T00:15:15.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.295+0000 7fb7e6c6e740 1 Processor -- start 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.295+0000 7fb7e6c6e740 1 -- start start 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.295+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 0x5556c0b72dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.295+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b7aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1ac830 con 0x5556c0b736a0 2024-03-09T00:15:15.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1aedd0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1ace30 con 0x5556c0b6aac0 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52714/0 (socket says 172.21.15.73:52714) 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 -- 172.21.15.73:0/2371251087 learned_addr learned my addr 172.21.15.73:0/2371251087 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 -- 172.21.15.73:0/2371251087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 msgr2=0x5556c0b7aae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 0x5556c0b72dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b7aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 -- 172.21.15.73:0/2371251087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 msgr2=0x5556c0b72dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 0x5556c0b72dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 -- 172.21.15.73:0/2371251087 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5556c0b6a8f0 con 0x5556c0b736a0 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 0x5556c0b72dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7e2622640 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4033b50 tx=0x7fb7d4042e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74e121b7fd56a752 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7c77fe640 1 -- 172.21.15.73:0/2371251087 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4003e40 con 0x5556c0b736a0 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7c77fe640 1 -- 172.21.15.73:0/2371251087 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d4003fe0 con 0x5556c0b736a0 2024-03-09T00:15:15.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.296+0000 7fb7c77fe640 1 -- 172.21.15.73:0/2371251087 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d4040690 con 0x5556c0b736a0 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2371251087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 msgr2=0x5556c0b6f670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4033b50 tx=0x7fb7d4042e30 comp rx=0 tx=0).stop 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2371251087 shutdown_connections 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b736a0 0x5556c0b6f670 unknown :-1 s=CLOSED pgs=927 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b6aac0 0x5556c0b7aae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/2371251087 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6a1a0 0x5556c0b72dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2371251087 >> 172.21.15.73:0/2371251087 conn(0x5556c0b648d0 msgr2=0x5556c0b6ef60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2371251087 shutdown_connections 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/2371251087 wait complete. 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 -- start start 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 Processor -- start 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.297+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0bd7cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b736a0 0x5556c0bd1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1ad450 con 0x5556c0b6a1a0 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1ae770 con 0x5556c0b6aac0 2024-03-09T00:15:15.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7dc1adab0 con 0x5556c0b736a0 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 10 client.0 _ll_register_callbacks cb 0x7fb7d80040a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52724/0 (socket says 172.21.15.73:52724) 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 -- 172.21.15.73:0/3238990775 learned_addr learned my addr 172.21.15.73:0/3238990775 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 -- 172.21.15.73:0/3238990775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b736a0 msgr2=0x5556c0bd1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b736a0 0x5556c0bd1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 -- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 msgr2=0x5556c0bd7cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0bd7cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 -- 172.21.15.73:0/3238990775 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5556c0bd2290 con 0x5556c0b6a1a0 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e1e21640 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80045a0 tx=0x7fb7d8004870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d71ea874df38eb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7c57fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7d802c060 con 0x5556c0b6a1a0 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb7d8027440 con 0x5556c0b6a1a0 2024-03-09T00:15:15.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb7d4003fe0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb7d8033050 con 0x5556c0b6a1a0 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 10 client.16986 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5556c0bdef60 con 0x5556c0b6a1a0 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.298+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7d4003fe0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb7d8011ca0 con 0x5556c0b6a1a0 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 10 client.16986 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 10 client.16986 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:15.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 20 client.16986 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0bdf500 0x5556c0be1940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 4 client.16986 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5556c0ad5750 con 0x5556c0bdf500 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0bdf500 0x5556c0be1940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7c57fa640 10 client.16986 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e1620640 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0bdf500 0x5556c0be1940 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3e1711b6c0bc86e3 server_cookie=66f3b724f92c918b in_seq=0 out_seq=0 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7c57fa640 1 -- 172.21.15.73:0/3238990775 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5556c0ad5750 con 0x5556c0bdf500 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7c57fa640 10 client.16986 handle_command_reply: tid=0 2024-03-09T00:15:15.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.299+0000 7fb7e6c6e740 1 client.16986 shutdown 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 msgr2=0x5556c0bd7790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 crypto rx=0x7fb7d80045a0 tx=0x7fb7d8004870 comp rx=0 tx=0).stop 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 shutdown_connections 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5556c0bdf500 0x5556c0be1940 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5556c0b736a0 0x5556c0bd1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5556c0b6aac0 0x5556c0bd7cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 --2- 172.21.15.73:0/3238990775 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5556c0b6a1a0 0x5556c0bd7790 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7c57fa640 0 client.16986 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:15.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 >> 172.21.15.73:0/3238990775 conn(0x5556c0b648d0 msgr2=0x5556c0b6f7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 shutdown_connections 2024-03-09T00:15:15.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 1 -- 172.21.15.73:0/3238990775 wait complete. 2024-03-09T00:15:15.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e6c6e740 20 client.16986 trim_cache size 0 max 16384 2024-03-09T00:15:15.314 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:15:15.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 msgr2=0x7fb7c80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 0x7fb7c80a4e30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fb7d4038040 tx=0x7fb7d4033340 comp rx=0 tx=0).stop 2024-03-09T00:15:15.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 msgr2=0x7fb7dc19dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:15.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 crypto rx=0x7fb7cc00d950 tx=0x7fb7cc00de10 comp rx=0 tx=0).stop 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 shutdown_connections 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb7dc105950 0x7fb7dc197950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb7c80a2970 0x7fb7c80a4e30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb7dc100300 0x7fb7dc19dce0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 --2- 172.21.15.73:0/3033378999 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb7dc0ff940 0x7fb7dc19d7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 >> 172.21.15.73:0/3033378999 conn(0x7fb7dc076710 msgr2=0x7fb7dc102710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:15.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 shutdown_connections 2024-03-09T00:15:15.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:15.300+0000 7fb7e4085640 1 -- 172.21.15.73:0/3033378999 wait complete. 2024-03-09T00:15:16.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:15 smithi102 ceph-mon[36106]: osdmap e100: 12 total, 12 up, 12 in 2024-03-09T00:15:16.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:15 smithi102 ceph-mon[36106]: pgmap v261: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:15:16.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:15 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/4113903046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:16.319 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:15:16.319 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:15:16.320 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:15:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:15 smithi179 ceph-mon[36637]: osdmap e100: 12 total, 12 up, 12 in 2024-03-09T00:15:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:15 smithi179 ceph-mon[36637]: pgmap v261: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:15:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:15 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/4113903046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:16.346 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:15 smithi073 ceph-mon[32309]: osdmap e100: 12 total, 12 up, 12 in 2024-03-09T00:15:16.346 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:15 smithi073 ceph-mon[32309]: pgmap v261: 129 pgs: 129 active+clean; 35 MiB data, 892 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:15:16.347 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:15 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/4113903046' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:16.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- 172.21.15.73:0/1257599602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001003f0 msgr2=0x7f39001007d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 --2- 172.21.15.73:0/1257599602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001003f0 0x7f39001007d0 secure :-1 s=READY pgs=929 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009900 tx=0x7f38f002bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- 172.21.15.73:0/1257599602 shutdown_connections 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 --2- 172.21.15.73:0/1257599602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f39001054c0 0x7f390010d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 --2- 172.21.15.73:0/1257599602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3900100da0 0x7f3900104de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 --2- 172.21.15.73:0/1257599602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001003f0 0x7f39001007d0 unknown :-1 s=CLOSED pgs=929 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.433 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- 172.21.15.73:0/1257599602 >> 172.21.15.73:0/1257599602 conn(0x7f39000772c0 msgr2=0x7f39000776d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- 172.21.15.73:0/1257599602 shutdown_connections 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- 172.21.15.73:0/1257599602 wait complete. 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 Processor -- start 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.432+0000 7f3908516640 1 -- start start 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 0x7f390019d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3900197ae0 0x7f3900197f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.434 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3900110650 con 0x7f39001054c0 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001104d0 con 0x7f3900197ae0 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001107d0 con 0x7f39001003f0 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52742/0 (socket says 172.21.15.73:52742) 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 -- 172.21.15.73:0/2565683735 learned_addr learned my addr 172.21.15.73:0/2565683735 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3906ab3640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3900197ae0 0x7f3900197f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.435 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f39062b2640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 0x7f390019d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 -- 172.21.15.73:0/2565683735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 msgr2=0x7f390019d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 0x7f390019d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 -- 172.21.15.73:0/2565683735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3900197ae0 msgr2=0x7f3900197f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3900197ae0 0x7f3900197f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 -- 172.21.15.73:0/2565683735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3900198600 con 0x7f39001054c0 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f39062b2640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 0x7f390019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3905ab1640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b450 tx=0x7f38fc00b910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f38f77fe640 1 -- 172.21.15.73:0/2565683735 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc015020 con 0x7f39001054c0 2024-03-09T00:15:16.436 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39001988f0 con 0x7f39001054c0 2024-03-09T00:15:16.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f38f77fe640 1 -- 172.21.15.73:0/2565683735 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc011d70 con 0x7f39001054c0 2024-03-09T00:15:16.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f38f77fe640 1 -- 172.21.15.73:0/2565683735 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc00f340 con 0x7f39001054c0 2024-03-09T00:15:16.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.433+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39001a4af0 con 0x7f39001054c0 2024-03-09T00:15:16.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.435+0000 7f38f77fe640 1 -- 172.21.15.73:0/2565683735 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f38fc018020 con 0x7f39001054c0 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 Processor -- start 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 -- start start 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6eae90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f8150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b64eb40 con 0x55cd8b6f8690 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b5f7d80 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f390b1ce740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b64abe0 con 0x55cd8b6eaab0 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f38f77fe640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 0x7f38ec0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.436+0000 7f3906ab3640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 0x7f38ec0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52748/0 (socket says 172.21.15.73:52748) 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 -- 172.21.15.73:0/1691418598 learned_addr learned my addr 172.21.15.73:0/1691418598 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6eae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f38f77fe640 1 -- 172.21.15.73:0/2565683735 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f38fc0cec50 con 0x7f39001054c0 2024-03-09T00:15:16.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f39062b2640 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f8150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 -- 172.21.15.73:0/1691418598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eaab0 msgr2=0x55cd8b6eae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6eae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 -- 172.21.15.73:0/1691418598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eb3d0 msgr2=0x55cd8b6f8150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f8150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 -- 172.21.15.73:0/1691418598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd8b42b7e0 con 0x55cd8b6f8690 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3906ab3640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 0x7f38ec0a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009fd0 tx=0x7f38f8009ee0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f38fc09acd0 tx=0x7f38fc06aaa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cfc1f429ce0577bd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f38f4bfe640 1 -- 172.21.15.73:0/1691418598 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc09b6d0 con 0x55cd8b6f8690 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f38f4bfe640 1 -- 172.21.15.73:0/1691418598 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc0661a0 con 0x55cd8b6f8690 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.437+0000 7f38f4bfe640 1 -- 172.21.15.73:0/1691418598 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc097640 con 0x55cd8b6f8690 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- 172.21.15.73:0/1691418598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 msgr2=0x55cd8b6faa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 crypto rx=0x7f38fc09acd0 tx=0x7f38fc06aaa0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- 172.21.15.73:0/1691418598 shutdown_connections 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6faa80 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f8150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1691418598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6eae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- 172.21.15.73:0/1691418598 >> 172.21.15.73:0/1691418598 conn(0x55cd8b6e54b0 msgr2=0x55cd8b6e5890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- 172.21.15.73:0/1691418598 shutdown_connections 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- 172.21.15.73:0/1691418598 wait complete. 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 -- start start 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.438+0000 7f390b1ce740 1 Processor -- start 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b6fb390 con 0x55cd8b6eaab0 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b60b1c0 con 0x55cd8b6f8690 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55cd8b6fb510 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 10 client.0 _ll_register_callbacks cb 0x55cd8b5fa7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52758/0 (socket says 172.21.15.73:52758) 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 -- 172.21.15.73:0/1901794772 learned_addr learned my addr 172.21.15.73:0/1901794772 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 -- 172.21.15.73:0/1901794772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 msgr2=0x55cd8b6f5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 -- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 msgr2=0x55cd8b6f5f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 -- 172.21.15.73:0/1901794772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd8b48bf70 con 0x55cd8b6eaab0 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:16.450 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f39062b2640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009230 tx=0x7f38f0004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=786cad326c0e35d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f38e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f0044070 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f38f00037f0 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38fc0661a0 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38f0050050 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 10 client.16998 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.439+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cd8b323170 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc0661a0 con 0x55cd8b6eaab0 2024-03-09T00:15:16.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f38f003e430 con 0x55cd8b6eaab0 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 10 client.16998 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 10 client.16998 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 20 client.16998 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b7086e0 0x55cd8b70ab10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 4 client.16998 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55cd8b624960 con 0x55cd8b7086e0 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b7086e0 0x55cd8b70ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b7086e0 0x55cd8b70ab10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f60c26cbd761697d server_cookie=eabcf4f28bfaa89 in_seq=0 out_seq=0 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.440+0000 7f38e9ffb640 10 client.16998 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1901794772 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55cd8b624960 con 0x55cd8b7086e0 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f38e9ffb640 10 client.16998 handle_command_reply: tid=0 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 client.16998 shutdown 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 msgr2=0x55cd8b6f54f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 secure :-1 s=READY pgs=932 cs=0 l=1 rev1=1 crypto rx=0x7f38f0009230 tx=0x7f38f0004270 comp rx=0 tx=0).stop 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 shutdown_connections 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b7086e0 0x55cd8b70ab10 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f38e9ffb640 0 client.16998 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1901794772 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f54f0 unknown :-1 s=CLOSED pgs=932 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 >> 172.21.15.73:0/1901794772 conn(0x55cd8b6e54b0 msgr2=0x55cd8b6e7200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 shutdown_connections 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.441+0000 7f390b1ce740 1 -- 172.21.15.73:0/1901794772 wait complete. 2024-03-09T00:15:16.454 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.442+0000 7f390b1ce740 20 client.16998 trim_cache size 0 max 16384 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.456+0000 7f390b1ce740 1 Processor -- start 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.456+0000 7f390b1ce740 1 -- start start 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f7c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001aa730 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001accd0 con 0x55cd8b6eaab0 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f390b1ce740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001aad30 con 0x55cd8b6f8690 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52760/0 (socket says 172.21.15.73:52760) 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 -- 172.21.15.73:0/240109982 learned_addr learned my addr 172.21.15.73:0/240109982 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f39062b2640 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f7c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 -- 172.21.15.73:0/240109982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6f8690 msgr2=0x55cd8b6f5bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 -- 172.21.15.73:0/240109982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 msgr2=0x55cd8b6f7c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f7c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 -- 172.21.15.73:0/240109982 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55cd8b6ebad0 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f3905ab1640 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0979a0 tx=0x7f38fc06cc00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4edbd062c72e3d1b server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f38ebfff640 1 -- 172.21.15.73:0/240109982 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0a3770 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f38ebfff640 1 -- 172.21.15.73:0/240109982 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc09afd0 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.457+0000 7f38ebfff640 1 -- 172.21.15.73:0/240109982 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38fc0a32e0 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- 172.21.15.73:0/240109982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 msgr2=0x55cd8b6f4670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 crypto rx=0x7f38fc0979a0 tx=0x7f38fc06cc00 comp rx=0 tx=0).stop 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- 172.21.15.73:0/240109982 shutdown_connections 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6f8690 0x55cd8b6f5bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b6f4670 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- 172.21.15.73:0/240109982 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b6f7c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- 172.21.15.73:0/240109982 >> 172.21.15.73:0/240109982 conn(0x55cd8b6e54b0 msgr2=0x55cd8b6f54b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- 172.21.15.73:0/240109982 shutdown_connections 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- 172.21.15.73:0/240109982 wait complete. 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- start start 2024-03-09T00:15:16.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 Processor -- start 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b758ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b759530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001ab350 con 0x55cd8b6f8690 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001ac670 con 0x55cd8b6eaab0 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39001ab9b0 con 0x55cd8b6eb3d0 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.458+0000 7f390b1ce740 10 client.0 _ll_register_callbacks cb 0x7f38fc010870 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:16.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52768/0 (socket says 172.21.15.73:52768) 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 -- 172.21.15.73:0/1193460414 learned_addr learned my addr 172.21.15.73:0/1193460414 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f39062b2640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b758ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3905ab1640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b759530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 -- 172.21.15.73:0/1193460414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 msgr2=0x55cd8b759530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b759530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 -- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 msgr2=0x55cd8b758ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b758ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 -- 172.21.15.73:0/1193460414 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55cd8b6ff180 con 0x55cd8b6f8690 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f39062b2640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b758ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f3906ab3640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f38f8008f50 tx=0x7f38f8004b50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6df8f914b16f3679 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f38e9ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38f802c060 con 0x55cd8b6f8690 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f38f8027650 con 0x55cd8b6f8690 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38fc09afd0 con 0x55cd8b6f8690 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38f8033050 con 0x55cd8b6f8690 2024-03-09T00:15:16.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f390b1ce740 10 client.17004 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55cd8b760860 con 0x55cd8b6f8690 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.459+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38fc09afd0 con 0x55cd8b6f8690 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f38f8025510 con 0x55cd8b6f8690 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 10 client.17004 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 10 client.17004 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 20 client.17004 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:16.468 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b760ff0 0x55cd8b763410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 4 client.17004 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55cd8b610080 con 0x55cd8b760ff0 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f39062b2640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b760ff0 0x55cd8b763410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f38e9ffb640 10 client.17004 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.460+0000 7f39062b2640 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b760ff0 0x55cd8b763410 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cea173c258eb622a server_cookie=2bd68d4ff6557964 in_seq=0 out_seq=0 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f38e9ffb640 1 -- 172.21.15.73:0/1193460414 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55cd8b610080 con 0x55cd8b760ff0 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f38e9ffb640 10 client.17004 handle_command_reply: tid=0 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f390b1ce740 1 client.17004 shutdown 2024-03-09T00:15:16.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 msgr2=0x55cd8b6fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 secure :-1 s=READY pgs=934 cs=0 l=1 rev1=1 crypto rx=0x7f38f8008f50 tx=0x7f38f8004b50 comp rx=0 tx=0).stop 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 shutdown_connections 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.461+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55cd8b760ff0 0x55cd8b763410 crc :-1 s=READY pgs=182 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55cd8b6f8690 0x55cd8b6fea90 unknown :-1 s=CLOSED pgs=934 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55cd8b6eb3d0 0x55cd8b759530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f38e9ffb640 0 client.17004 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 --2- 172.21.15.73:0/1193460414 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55cd8b6eaab0 0x55cd8b758ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 >> 172.21.15.73:0/1193460414 conn(0x55cd8b6e54b0 msgr2=0x55cd8b6f44f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 shutdown_connections 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 1 -- 172.21.15.73:0/1193460414 wait complete. 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f390b1ce740 20 client.17004 trim_cache size 0 max 16384 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:16.471 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:15:16.472 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "5fcd583e-d4fb-4207-b1e8-32c1caf0c2c2", 2024-03-09T00:15:16.472 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:15:16.472 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:16.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 msgr2=0x7f38ec0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 0x7f38ec0a4e30 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f38f8009fd0 tx=0x7f38f8009ee0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 msgr2=0x7f390019de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 crypto rx=0x7f38fc00b450 tx=0x7f38fc00b910 comp rx=0 tx=0).stop 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 shutdown_connections 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3900197ae0 0x7f3900197f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38ec0a2970 0x7f38ec0a4e30 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f39001054c0 0x7f390019de70 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 --2- 172.21.15.73:0/2565683735 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f39001003f0 0x7f390019d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 >> 172.21.15.73:0/2565683735 conn(0x7f39000772c0 msgr2=0x7f3900069c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 shutdown_connections 2024-03-09T00:15:16.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.462+0000 7f3908516640 1 -- 172.21.15.73:0/2565683735 wait complete. 2024-03-09T00:15:16.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:15:16.475 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:15:16.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 -- 172.21.15.73:0/699430802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 msgr2=0x7f5478106650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 --2- 172.21.15.73:0/699430802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f5478106650 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 crypto rx=0x7f5468009900 tx=0x7f546802bcd0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 -- 172.21.15.73:0/699430802 shutdown_connections 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 --2- 172.21.15.73:0/699430802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 0x7f547810d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 --2- 172.21.15.73:0/699430802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 0x7f5478100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 --2- 172.21.15.73:0/699430802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f5478106650 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 -- 172.21.15.73:0/699430802 >> 172.21.15.73:0/699430802 conn(0x7f5478077280 msgr2=0x7f5478077690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 -- 172.21.15.73:0/699430802 shutdown_connections 2024-03-09T00:15:16.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 -- 172.21.15.73:0/699430802 wait complete. 2024-03-09T00:15:16.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.629+0000 7f547e092640 1 Processor -- start 2024-03-09T00:15:16.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 -- start start 2024-03-09T00:15:16.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 0x7f5478101e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 0x7f54781023a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781105a0 con 0x7f5478106270 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478110420 con 0x7f5478100b50 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f547e092640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5478110720 con 0x7f54781001b0 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52790/0 (socket says 172.21.15.73:52790) 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 -- 172.21.15.73:0/3742495291 learned_addr learned my addr 172.21.15.73:0/3742495291 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f54777fe640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 0x7f5478101e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5476ffd640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 0x7f54781023a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 -- 172.21.15.73:0/3742495291 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 msgr2=0x7f5478101e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 0x7f5478101e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 -- 172.21.15.73:0/3742495291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 msgr2=0x7f54781023a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 0x7f54781023a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 -- 172.21.15.73:0/3742495291 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54781a5ff0 con 0x7f5478106270 2024-03-09T00:15:16.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.630+0000 7f5477fff640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f547000eb10 tx=0x7f547000c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.631+0000 7f5474ff9640 1 -- 172.21.15.73:0/3742495291 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470004b20 con 0x7f5478106270 2024-03-09T00:15:16.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.631+0000 7f5474ff9640 1 -- 172.21.15.73:0/3742495291 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5470004cc0 con 0x7f5478106270 2024-03-09T00:15:16.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.631+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54781a62e0 con 0x7f5478106270 2024-03-09T00:15:16.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.631+0000 7f5474ff9640 1 -- 172.21.15.73:0/3742495291 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5470010760 con 0x7f5478106270 2024-03-09T00:15:16.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.631+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54781a6830 con 0x7f5478106270 2024-03-09T00:15:16.643 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5474ff9640 1 -- 172.21.15.73:0/3742495291 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f5470005020 con 0x7f5478106270 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 Processor -- start 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 -- start start 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a89bc60 0x56554a89e050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a7f5e00 con 0x56554a892760 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a79eb10 con 0x56554a89bc60 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.633+0000 7f5480c71740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a7f1ea0 con 0x56554a893080 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5474ff9640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 0x7f54600a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f54777fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52798/0 (socket says 172.21.15.73:52798) 2024-03-09T00:15:16.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 -- 172.21.15.73:0/606213109 learned_addr learned my addr 172.21.15.73:0/606213109 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f54777fe640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:39458/0 (socket says 172.21.15.73:39458) 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a89bc60 0x56554a89e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5477fff640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 0x7f54600a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5474ff9640 1 -- 172.21.15.73:0/3742495291 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f54700cf700 con 0x7f5478106270 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 -- 172.21.15.73:0/606213109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 msgr2=0x56554a89b720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 -- 172.21.15.73:0/606213109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a89bc60 msgr2=0x56554a89e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a89bc60 0x56554a89e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 -- 172.21.15.73:0/606213109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56554a5d47d0 con 0x56554a892760 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f54777fe640 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5476ffd640 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f546c013630 tx=0x7f546c009780 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3d466246baa37fad server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f5477fff640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 0x7f54600a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f547009b670 tx=0x7f5470086040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f545dffb640 1 -- 172.21.15.73:0/606213109 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c004050 con 0x56554a892760 2024-03-09T00:15:16.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f545dffb640 1 -- 172.21.15.73:0/606213109 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c0041f0 con 0x56554a892760 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.634+0000 7f545dffb640 1 -- 172.21.15.73:0/606213109 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c00f7e0 con 0x56554a892760 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 -- 172.21.15.73:0/606213109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 msgr2=0x56554a892b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 secure :-1 s=READY pgs=937 cs=0 l=1 rev1=1 crypto rx=0x7f546c013630 tx=0x7f546c009780 comp rx=0 tx=0).stop 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 -- 172.21.15.73:0/606213109 shutdown_connections 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a89bc60 0x56554a89e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a893080 0x56554a89b720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 --2- 172.21.15.73:0/606213109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a892b40 unknown :-1 s=CLOSED pgs=937 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.648 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 -- 172.21.15.73:0/606213109 >> 172.21.15.73:0/606213109 conn(0x56554a88ce90 msgr2=0x56554a88d270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 -- 172.21.15.73:0/606213109 shutdown_connections 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.635+0000 7f5480c71740 1 -- 172.21.15.73:0/606213109 wait complete. 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 -- start start 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 Processor -- start 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a898730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.649 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a898c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a89ea50 con 0x56554a892760 2024-03-09T00:15:16.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a7b20a0 con 0x56554a893080 2024-03-09T00:15:16.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56554a89ebd0 con 0x56554a89bc60 2024-03-09T00:15:16.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5480c71740 10 client.0 _ll_register_callbacks cb 0x56554a7b1a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:16.650 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f54777fe640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a898730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52812/0 (socket says 172.21.15.73:52812) 2024-03-09T00:15:16.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5476ffd640 1 -- 172.21.15.73:0/23978520 learned_addr learned my addr 172.21.15.73:0/23978520 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.636+0000 7f5476ffd640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a898c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 -- 172.21.15.73:0/23978520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 msgr2=0x56554a898c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a898c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 -- 172.21.15.73:0/23978520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 msgr2=0x56554a89b3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 -- 172.21.15.73:0/23978520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56554a50a3f0 con 0x56554a893080 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5476ffd640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f54777fe640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a898730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5468036040 tx=0x7f54680045b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0798428b1c3af6a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.102:3300/0 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468043070 con 0x56554a893080 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f546c0041f0 con 0x56554a893080 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mon.2 v2:172.21.15.102:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f546802cb30 con 0x56554a893080 2024-03-09T00:15:16.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5468050050 con 0x56554a893080 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5480c71740 10 client.26177 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.637+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56554a4ca0f0 con 0x56554a893080 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mon.2 v2:172.21.15.102:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f546c0041f0 con 0x56554a893080 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mon.2 v2:172.21.15.102:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f5468005210 con 0x56554a893080 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 10 client.26177 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:16.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 10 client.26177 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:16.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 20 client.26177 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:16.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a8ab690 0x56554a8ada80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 4 client.26177 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:16.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56554a803340 con 0x56554a8ab690 2024-03-09T00:15:16.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.638+0000 7f5476ffd640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a8ab690 0x56554a8ada80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5476ffd640 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a8ab690 0x56554a8ada80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fdd4dd7d4ce31548 server_cookie=55d70793d90a4b6e in_seq=0 out_seq=0 2024-03-09T00:15:16.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5456ffd640 10 client.26177 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5456ffd640 1 -- 172.21.15.73:0/23978520 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56554a803340 con 0x56554a8ab690 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5456ffd640 10 client.26177 handle_command_reply: tid=0 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 client.26177 shutdown 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 msgr2=0x56554a898730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a898730 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f5468036040 tx=0x7f54680045b0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 shutdown_connections 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a8ab690 0x56554a8ada80 crc :-1 s=READY pgs=183 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a898c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a898730 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5456ffd640 0 client.26177 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 --2- 172.21.15.73:0/23978520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89b3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.639+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 >> 172.21.15.73:0/23978520 conn(0x56554a88ce90 msgr2=0x56554a88ebe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.640+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 shutdown_connections 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.640+0000 7f5480c71740 1 -- 172.21.15.73:0/23978520 wait complete. 2024-03-09T00:15:16.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.640+0000 7f5480c71740 20 client.26177 trim_cache size 0 max 16384 2024-03-09T00:15:16.667 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 Processor -- start 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 -- start start 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a892760 0x56554a89ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8a2e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781aea00 con 0x56554a893080 2024-03-09T00:15:16.668 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781b0fa0 con 0x56554a892760 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5480c71740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781af000 con 0x56554a89bc60 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f54777fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f54777fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52822/0 (socket says 172.21.15.73:52822) 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f54777fe640 1 -- 172.21.15.73:0/1662811249 learned_addr learned my addr 172.21.15.73:0/1662811249 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.657+0000 7f5476ffd640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a892760 0x56554a89ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f5476ffd640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8a2e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.669 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 -- 172.21.15.73:0/1662811249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 msgr2=0x56554a8a2e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8a2e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 -- 172.21.15.73:0/1662811249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a892760 msgr2=0x56554a89ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a892760 0x56554a89ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 -- 172.21.15.73:0/1662811249 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56554a896c80 con 0x56554a893080 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f5476ffd640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:16.670 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f54777fe640 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f54680336e0 tx=0x7f5468001e70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bb928ddeb6f8ed0e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f545d7fa640 1 -- 172.21.15.73:0/1662811249 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5468043040 con 0x56554a893080 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f545d7fa640 1 -- 172.21.15.73:0/1662811249 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5468005730 con 0x56554a893080 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.658+0000 7f545d7fa640 1 -- 172.21.15.73:0/1662811249 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546803de60 con 0x56554a893080 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- 172.21.15.73:0/1662811249 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 msgr2=0x56554a89b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 crypto rx=0x7f54680336e0 tx=0x7f5468001e70 comp rx=0 tx=0).stop 2024-03-09T00:15:16.671 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- 172.21.15.73:0/1662811249 shutdown_connections 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8a2e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a893080 0x56554a89b3a0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 --2- 172.21.15.73:0/1662811249 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a892760 0x56554a89ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- 172.21.15.73:0/1662811249 >> 172.21.15.73:0/1662811249 conn(0x56554a88ce90 msgr2=0x56554a8b2340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- 172.21.15.73:0/1662811249 shutdown_connections 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- 172.21.15.73:0/1662811249 wait complete. 2024-03-09T00:15:16.672 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 -- start start 2024-03-09T00:15:16.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.659+0000 7f5480c71740 1 Processor -- start 2024-03-09T00:15:16.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a89f5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.673 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8faaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781af620 con 0x56554a892760 2024-03-09T00:15:16.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781b0940 con 0x56554a893080 2024-03-09T00:15:16.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54781afc80 con 0x56554a89bc60 2024-03-09T00:15:16.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 10 client.0 _ll_register_callbacks cb 0x7f546c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:16.674 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:52830/0 (socket says 172.21.15.73:52830) 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 -- 172.21.15.73:0/3010931331 learned_addr learned my addr 172.21.15.73:0/3010931331 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f54777fe640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a89f5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8faaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 -- 172.21.15.73:0/3010931331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 msgr2=0x56554a8faaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8faaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.675 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 -- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 msgr2=0x56554a89f5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a89f5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 -- 172.21.15.73:0/3010931331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56554a8fb0b0 con 0x56554a892760 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5476ffd640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f546c0058c0 tx=0x7f546c009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f5274aec78b67b77 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5456ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546c034060 con 0x56554a892760 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f546c02ea90 con 0x56554a892760 2024-03-09T00:15:16.676 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.660+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5468005730 con 0x56554a892760 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f546c003b10 con 0x56554a892760 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 10 client.17022 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56554a902470 con 0x56554a892760 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5468005730 con 0x56554a892760 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f546c00fa70 con 0x56554a892760 2024-03-09T00:15:16.677 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 10 client.17022 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:16.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 10 client.17022 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:16.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 20 client.17022 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:16.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a902980 0x56554a904da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:16.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 4 client.17022 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:15:16.678 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56554a7fd690 con 0x56554a902980 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.661+0000 7f54777fe640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a902980 0x56554a904da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5456ffd640 10 client.17022 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f54777fe640 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a902980 0x56554a904da0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7b641d63e624314 server_cookie=2974a92d2e4b9bd4 in_seq=0 out_seq=0 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5456ffd640 1 -- 172.21.15.73:0/3010931331 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x56554a7fd690 con 0x56554a902980 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5456ffd640 10 client.17022 handle_command_reply: tid=0 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 client.17022 shutdown 2024-03-09T00:15:16.679 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 msgr2=0x56554a89f0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 crypto rx=0x7f546c0058c0 tx=0x7f546c009940 comp rx=0 tx=0).stop 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 shutdown_connections 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56554a902980 0x56554a904da0 crc :-1 s=READY pgs=184 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56554a89bc60 0x56554a8faaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56554a893080 0x56554a89f5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5456ffd640 0 client.17022 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:16.680 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 --2- 172.21.15.73:0/3010931331 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56554a892760 0x56554a89f0b0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 >> 172.21.15.73:0/3010931331 conn(0x56554a88ce90 msgr2=0x56554a8986d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.662+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 shutdown_connections 2024-03-09T00:15:16.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f5480c71740 1 -- 172.21.15.73:0/3010931331 wait complete. 2024-03-09T00:15:16.681 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f5480c71740 20 client.17022 trim_cache size 0 max 16384 2024-03-09T00:15:16.681 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2020 inodes in the stack)", 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "5fcd583e-d4fb-4207-b1e8-32c1caf0c2c2": { 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "5fcd583e-d4fb-4207-b1e8-32c1caf0c2c2", 2024-03-09T00:15:16.682 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:15:16.683 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:15:16.683 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:15:16.683 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:16.684 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 msgr2=0x7f54600a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 0x7f54600a4eb0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f547009b670 tx=0x7f5470086040 comp rx=0 tx=0).stop 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 msgr2=0x7f54781028e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 crypto rx=0x7f547000eb10 tx=0x7f547000c2d0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 shutdown_connections 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f54600a29f0 0x7f54600a4eb0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f5478106270 0x7f54781028e0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f5478100b50 0x7f54781023a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 --2- 172.21.15.73:0/3742495291 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f54781001b0 0x7f5478101e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:16.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 >> 172.21.15.73:0/3742495291 conn(0x7f5478077280 msgr2=0x7f54780ff370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:16.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 shutdown_connections 2024-03-09T00:15:16.686 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:16.663+0000 7f547e092640 1 -- 172.21.15.73:0/3742495291 wait complete. 2024-03-09T00:15:16.686 INFO:tasks.cephfs.filesystem:scrub status for tag:5fcd583e-d4fb-4207-b1e8-32c1caf0c2c2 - {'path': '/', 'tag': '5fcd583e-d4fb-4207-b1e8-32c1caf0c2c2', 'options': 'recursive,force'} 2024-03-09T00:15:17.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:17 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:15:17.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:17 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:17.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:17 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:15:17.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:17 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:17.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:17 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:15:17.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:17 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:17.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:17 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:15:17.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:17 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:17.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:17 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:15:17.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:17 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:17.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:17 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:15:17.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:17 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:18.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:18 smithi179 ceph-mon[36637]: pgmap v262: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 255 B/s wr, 272 op/s 2024-03-09T00:15:18.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:18 smithi073 ceph-mon[32309]: pgmap v262: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 255 B/s wr, 272 op/s 2024-03-09T00:15:18.413 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:18 smithi102 ceph-mon[36106]: pgmap v262: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 272 KiB/s rd, 255 B/s wr, 272 op/s 2024-03-09T00:15:19.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:19 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:15:19.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:19 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:15:19.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:19 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:15:20.334 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:20 smithi179 ceph-mon[36637]: pgmap v263: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 206 B/s wr, 348 op/s 2024-03-09T00:15:20.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:20 smithi073 ceph-mon[32309]: pgmap v263: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 206 B/s wr, 348 op/s 2024-03-09T00:15:20.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:20 smithi102 ceph-mon[36106]: pgmap v263: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 348 KiB/s rd, 206 B/s wr, 348 op/s 2024-03-09T00:15:22.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:22 smithi102 ceph-mon[36106]: pgmap v264: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:15:22.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:22 smithi179 ceph-mon[36637]: pgmap v264: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:15:22.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:22 smithi073 ceph-mon[32309]: pgmap v264: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 204 B/s wr, 346 op/s 2024-03-09T00:15:24.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:24 smithi102 ceph-mon[36106]: pgmap v265: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:15:24.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:24 smithi179 ceph-mon[36637]: pgmap v265: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:15:24.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:24 smithi073 ceph-mon[32309]: pgmap v265: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:15:26.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:26 smithi102 ceph-mon[36106]: pgmap v266: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:15:26.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:26 smithi179 ceph-mon[36637]: pgmap v266: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:15:26.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:26 smithi073 ceph-mon[32309]: pgmap v266: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:15:28.457 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:28 smithi073 ceph-mon[32309]: pgmap v267: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:15:28.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:28 smithi102 ceph-mon[36106]: pgmap v267: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:15:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:28 smithi179 ceph-mon[36637]: pgmap v267: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:15:30.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:30 smithi102 ceph-mon[36106]: pgmap v268: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:15:30.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:30 smithi179 ceph-mon[36637]: pgmap v268: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:15:30.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:30 smithi073 ceph-mon[32309]: pgmap v268: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 107 KiB/s rd, 108 op/s 2024-03-09T00:15:32.516 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:32 smithi102 ceph-mon[36106]: pgmap v269: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:32.584 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:32 smithi179 ceph-mon[36637]: pgmap v269: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:32.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:32 smithi073 ceph-mon[32309]: pgmap v269: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:33.515 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:33 smithi102 ceph-mon[36106]: pgmap v270: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:33.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:33 smithi179 ceph-mon[36637]: pgmap v270: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:33.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:33 smithi073 ceph-mon[32309]: pgmap v270: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:36.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:36 smithi102 ceph-mon[36106]: pgmap v271: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:36.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:36 smithi179 ceph-mon[36637]: pgmap v271: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:36.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:36 smithi073 ceph-mon[32309]: pgmap v271: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:38.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:38 smithi102 ceph-mon[36106]: pgmap v272: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:38.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:38 smithi179 ceph-mon[36637]: pgmap v272: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:38.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:38 smithi073 ceph-mon[32309]: pgmap v272: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:40.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:40 smithi102 ceph-mon[36106]: pgmap v273: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:40.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:40 smithi179 ceph-mon[36637]: pgmap v273: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:40.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:40 smithi073 ceph-mon[32309]: pgmap v273: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:42.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:42 smithi102 ceph-mon[36106]: pgmap v274: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:42.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:42 smithi179 ceph-mon[36637]: pgmap v274: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:42.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:42 smithi073 ceph-mon[32309]: pgmap v274: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:44.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:44 smithi102 ceph-mon[36106]: pgmap v275: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:44.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:44 smithi179 ceph-mon[36637]: pgmap v275: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:44.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:44 smithi073 ceph-mon[32309]: pgmap v275: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:46.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:45 smithi102 ceph-mon[36106]: pgmap v276: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:46.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:45 smithi179 ceph-mon[36637]: pgmap v276: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:46.404 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:45 smithi073 ceph-mon[32309]: pgmap v276: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:46.687 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:15:46.687 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:15:46.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 -- 172.21.15.73:0/1289804728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6984101ea0 msgr2=0x7f6984105ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 --2- 172.21.15.73:0/1289804728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6984101ea0 0x7f6984105ee0 secure :-1 s=READY pgs=940 cs=0 l=1 rev1=1 crypto rx=0x7f6980009900 tx=0x7f698002bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:46.806 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 -- 172.21.15.73:0/1289804728 shutdown_connections 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 --2- 172.21.15.73:0/1289804728 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 0x7f698410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 --2- 172.21.15.73:0/1289804728 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f6984101ea0 0x7f6984105ee0 unknown :-1 s=CLOSED pgs=940 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 --2- 172.21.15.73:0/1289804728 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f69841014f0 0x7f69841018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 -- 172.21.15.73:0/1289804728 >> 172.21.15.73:0/1289804728 conn(0x7f69840786f0 msgr2=0x7f6984078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 -- 172.21.15.73:0/1289804728 shutdown_connections 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.805+0000 7f698b859640 1 -- 172.21.15.73:0/1289804728 wait complete. 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 Processor -- start 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 -- start start 2024-03-09T00:15:46.807 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 0x7f6984073b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 0x7f69841a5d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6984111650 con 0x7f69841014f0 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841114d0 con 0x7f69841065c0 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f698b859640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841117d0 con 0x7f6984074070 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34144/0 (socket says 172.21.15.73:34144) 2024-03-09T00:15:46.808 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f69895f5640 1 -- 172.21.15.73:0/2772595918 learned_addr learned my addr 172.21.15.73:0/2772595918 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.806+0000 7f6988df4640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 0x7f6984073b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f6989df6640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 0x7f69841a5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 -- 172.21.15.73:0/2772595918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 msgr2=0x7f69841a5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 0x7f69841a5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 -- 172.21.15.73:0/2772595918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 msgr2=0x7f6984073b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 0x7f6984073b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 -- 172.21.15.73:0/2772595918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69841a6480 con 0x7f69841014f0 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f6989df6640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 0x7f69841a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:46.809 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f69895f5640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f697400d770 tx=0x7f697400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f697a7fc640 1 -- 172.21.15.73:0/2772595918 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974017070 con 0x7f69841014f0 2024-03-09T00:15:46.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69841a6770 con 0x7f69841014f0 2024-03-09T00:15:46.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f697a7fc640 1 -- 172.21.15.73:0/2772595918 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6974005500 con 0x7f69841014f0 2024-03-09T00:15:46.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f697a7fc640 1 -- 172.21.15.73:0/2772595918 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974004e00 con 0x7f69841014f0 2024-03-09T00:15:46.810 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.807+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69841a6c40 con 0x7f69841014f0 2024-03-09T00:15:46.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.810+0000 7f697a7fc640 1 -- 172.21.15.73:0/2772595918 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6974004fa0 con 0x7f69841014f0 2024-03-09T00:15:46.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.810+0000 7f698e43b740 1 Processor -- start 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.810+0000 7f698e43b740 1 -- start start 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb6a720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f697a7fc640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 0x7f69700a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb6d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606eac4e00 con 0x55606eb61760 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606ea6db10 con 0x55606eb6ac60 2024-03-09T00:15:46.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f698e43b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606eac0ea0 con 0x55606eb62080 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34148/0 (socket says 172.21.15.73:34148) 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 -- 172.21.15.73:0/3006115738 learned_addr learned my addr 172.21.15.73:0/3006115738 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6988df4640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 0x7f69700a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f697a7fc640 1 -- 172.21.15.73:0/2772595918 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f69740cece0 con 0x7f69841014f0 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb6d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f69895f5640 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb6a720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 -- 172.21.15.73:0/3006115738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 msgr2=0x55606eb6a720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb6a720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 -- 172.21.15.73:0/3006115738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 msgr2=0x55606eb6d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb6d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 -- 172.21.15.73:0/3006115738 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55606e8a37d0 con 0x55606eb61760 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6989df6640 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f697c013820 tx=0x7f697c009a90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e37712412ec6453f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f6988df4640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 0x7f69700a4e60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6980009230 tx=0x7f6980005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f696f7fe640 1 -- 172.21.15.73:0/3006115738 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c019020 con 0x55606eb61760 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f696f7fe640 1 -- 172.21.15.73:0/3006115738 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f697c011d60 con 0x55606eb61760 2024-03-09T00:15:46.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.811+0000 7f696f7fe640 1 -- 172.21.15.73:0/3006115738 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00f810 con 0x55606eb61760 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- 172.21.15.73:0/3006115738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 msgr2=0x55606eb61b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 crypto rx=0x7f697c013820 tx=0x7f697c009a90 comp rx=0 tx=0).stop 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- 172.21.15.73:0/3006115738 shutdown_connections 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb6d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb6a720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 --2- 172.21.15.73:0/3006115738 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb61b40 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- 172.21.15.73:0/3006115738 >> 172.21.15.73:0/3006115738 conn(0x55606eb5be90 msgr2=0x55606eb5c270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.822 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- 172.21.15.73:0/3006115738 shutdown_connections 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- 172.21.15.73:0/3006115738 wait complete. 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.812+0000 7f698e43b740 1 -- start start 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 Processor -- start 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb67770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb67cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606eb6da50 con 0x55606eb61760 2024-03-09T00:15:46.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606ea810a0 con 0x55606eb6ac60 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55606eb6dbd0 con 0x55606eb62080 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f698e43b740 10 client.0 _ll_register_callbacks cb 0x55606ea80a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34160/0 (socket says 172.21.15.73:34160) 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 -- 172.21.15.73:0/1074244602 learned_addr learned my addr 172.21.15.73:0/1074244602 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f6989df6640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb67cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f6988df4640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb67770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 -- 172.21.15.73:0/1074244602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 msgr2=0x55606eb67770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb67770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 -- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 msgr2=0x55606eb67cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb67cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f69895f5640 1 -- 172.21.15.73:0/1074244602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55606e7d93f0 con 0x55606eb61760 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f6988df4640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb67770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.813+0000 7f6989df6640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb67cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f69895f5640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f697409ae80 tx=0x7f6974004800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ec753860d5bfb3b9 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:46.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974020040 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f6974097560 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f697c011d60 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f697409f040 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 10 client.17034 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55606e7990f0 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f697c011d60 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f697409a340 con 0x55606eb61760 2024-03-09T00:15:46.826 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 10 client.17034 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 10 client.17034 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 20 client.17034 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606eb7a670 0x55606eb7caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 4 client.17034 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.814+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55606ead2340 con 0x55606eb7a670 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.815+0000 7f6988df4640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606eb7a670 0x55606eb7caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.815+0000 7f696cff9640 10 client.17034 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:46.827 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.815+0000 7f6988df4640 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606eb7a670 0x55606eb7caa0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d4e4bc609699f8a server_cookie=14c34ba4f029a3a5 in_seq=0 out_seq=0 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f696cff9640 1 -- 172.21.15.73:0/1074244602 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55606ead2340 con 0x55606eb7a670 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f696cff9640 10 client.17034 handle_command_reply: tid=0 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 client.17034 shutdown 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 msgr2=0x55606eb6a3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 crypto rx=0x7f697409ae80 tx=0x7f6974004800 comp rx=0 tx=0).stop 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 shutdown_connections 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606eb7a670 0x55606eb7caa0 crc :-1 s=READY pgs=185 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb67cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.828 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f696cff9640 0 client.17034 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb67770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 --2- 172.21.15.73:0/1074244602 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb6a3e0 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 >> 172.21.15.73:0/1074244602 conn(0x55606eb5be90 msgr2=0x55606eb5dbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 shutdown_connections 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 1 -- 172.21.15.73:0/1074244602 wait complete. 2024-03-09T00:15:46.829 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.816+0000 7f698e43b740 20 client.17034 trim_cache size 0 max 16384 2024-03-09T00:15:46.835 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 Processor -- start 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 -- start start 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb5ad70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb69ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841aea00 con 0x55606eb61760 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841b0fa0 con 0x55606eb6ac60 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f698e43b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841af000 con 0x55606eb62080 2024-03-09T00:15:46.836 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34166/0 (socket says 172.21.15.73:34166) 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.831+0000 7f69895f5640 1 -- 172.21.15.73:0/738024142 learned_addr learned my addr 172.21.15.73:0/738024142 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f6988df4640 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb5ad70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 -- 172.21.15.73:0/738024142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 msgr2=0x55606eb5ad70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb5ad70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 -- 172.21.15.73:0/738024142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 msgr2=0x55606eb69ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.837 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb69ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 -- 172.21.15.73:0/738024142 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55606eb61eb0 con 0x55606eb61760 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f69895f5640 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f697409aba0 tx=0x7f69740048a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=89c8a70e017eccec server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f696effd640 1 -- 172.21.15.73:0/738024142 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6974020050 con 0x55606eb61760 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f696effd640 1 -- 172.21.15.73:0/738024142 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69740a3690 con 0x55606eb61760 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f696effd640 1 -- 172.21.15.73:0/738024142 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697409a4d0 con 0x55606eb61760 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f698e43b740 1 -- 172.21.15.73:0/738024142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 msgr2=0x55606eb67050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.832+0000 7f698e43b740 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 secure :-1 s=READY pgs=944 cs=0 l=1 rev1=1 crypto rx=0x7f697409aba0 tx=0x7f69740048a0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.838 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- 172.21.15.73:0/738024142 shutdown_connections 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb6ac60 0x55606eb69ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb62080 0x55606eb5ad70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- 172.21.15.73:0/738024142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb67050 unknown :-1 s=CLOSED pgs=944 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- 172.21.15.73:0/738024142 >> 172.21.15.73:0/738024142 conn(0x55606eb5be90 msgr2=0x55606eb5a660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- 172.21.15.73:0/738024142 shutdown_connections 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- 172.21.15.73:0/738024142 wait complete. 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- start start 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 Processor -- start 2024-03-09T00:15:46.839 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb62080 0x55606eb704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb6ac60 0x55606eb70a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841af620 con 0x55606eb61760 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841b0940 con 0x55606eb62080 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69841afc80 con 0x55606eb6ac60 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f698e43b740 10 client.0 _ll_register_callbacks cb 0x7f6974097190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.833+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.840 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34182/0 (socket says 172.21.15.73:34182) 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 -- 172.21.15.73:0/675138161 learned_addr learned my addr 172.21.15.73:0/675138161 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f6989df6640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb6ac60 0x55606eb70a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f6988df4640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb62080 0x55606eb704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 -- 172.21.15.73:0/675138161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb6ac60 msgr2=0x55606eb70a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb6ac60 0x55606eb70a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 -- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb62080 msgr2=0x55606eb704e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb62080 0x55606eb704e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 -- 172.21.15.73:0/675138161 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55606eb71250 con 0x55606eb61760 2024-03-09T00:15:46.841 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f69895f5640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f69740b8e50 tx=0x7f697409a6e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34263cd8a43118df server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f696cff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f69740aa060 con 0x55606eb61760 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f69740a3690 con 0x55606eb61760 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f69740840d0 con 0x55606eb61760 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6974020050 con 0x55606eb61760 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f698e43b740 10 client.17040 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55606ebd1bd0 con 0x55606eb61760 2024-03-09T00:15:46.842 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.834+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f69740a3690 con 0x55606eb61760 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f69740a0080 con 0x55606eb61760 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 10 client.17040 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 10 client.17040 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 20 client.17040 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606ebd2000 0x55606ebd43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 4 client.17040 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55606eacc690 con 0x55606ebd2000 2024-03-09T00:15:46.843 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f6988df4640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606ebd2000 0x55606ebd43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f696cff9640 10 client.17040 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f6988df4640 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606ebd2000 0x55606ebd43f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d98c8b49cdfb7de server_cookie=835d15411acaba14 in_seq=0 out_seq=0 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f696cff9640 1 -- 172.21.15.73:0/675138161 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55606eacc690 con 0x55606ebd2000 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f696cff9640 10 client.17040 handle_command_reply: tid=0 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.835+0000 7f698e43b740 1 client.17040 shutdown 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 msgr2=0x55606eb72140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 crypto rx=0x7f69740b8e50 tx=0x7f697409a6e0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.844 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 shutdown_connections 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55606ebd2000 0x55606ebd43f0 crc :-1 s=READY pgs=186 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55606eb6ac60 0x55606eb70a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55606eb62080 0x55606eb704e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f696cff9640 0 client.17040 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 --2- 172.21.15.73:0/675138161 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55606eb61760 0x55606eb72140 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 >> 172.21.15.73:0/675138161 conn(0x55606eb5be90 msgr2=0x55606eb5aef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 shutdown_connections 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 1 -- 172.21.15.73:0/675138161 wait complete. 2024-03-09T00:15:46.845 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698e43b740 20 client.17040 trim_cache size 0 max 16384 2024-03-09T00:15:46.846 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:46.846 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:15:46.846 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:15:46.846 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 msgr2=0x7f69700a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 0x7f69700a4e60 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6980009230 tx=0x7f6980005fd0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 msgr2=0x7f69840735f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 crypto rx=0x7f697400d770 tx=0x7f697400dc30 comp rx=0 tx=0).stop 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 shutdown_connections 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f6984074070 0x7f69841a5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f69841065c0 0x7f6984073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.848 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f69700a29a0 0x7f69700a4e60 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 --2- 172.21.15.73:0/2772595918 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f69841014f0 0x7f69840735f0 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 >> 172.21.15.73:0/2772595918 conn(0x7f69840786f0 msgr2=0x7f69840ff440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 shutdown_connections 2024-03-09T00:15:46.849 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.836+0000 7f698b859640 1 -- 172.21.15.73:0/2772595918 wait complete. 2024-03-09T00:15:46.849 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:15:46.850 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:15:46.953 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 -- 172.21.15.73:0/1611678888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 msgr2=0x7f9e38102940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 --2- 172.21.15.73:0/1611678888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e38102940 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 crypto rx=0x7f9e28009900 tx=0x7f9e2802bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:46.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 -- 172.21.15.73:0/1611678888 shutdown_connections 2024-03-09T00:15:46.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 --2- 172.21.15.73:0/1611678888 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 0x7f9e3810ddd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 --2- 172.21.15.73:0/1611678888 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 0x7f9e3810b360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.954 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 --2- 172.21.15.73:0/1611678888 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e38102940 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 -- 172.21.15.73:0/1611678888 >> 172.21.15.73:0/1611678888 conn(0x7f9e380772f0 msgr2=0x7f9e38077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 -- 172.21.15.73:0/1611678888 shutdown_connections 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.953+0000 7f9e3d774640 1 -- 172.21.15.73:0/1611678888 wait complete. 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 Processor -- start 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 -- start start 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 0x7f9e381006f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 0x7f9e38100c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.955 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38111710 con 0x7f9e38102560 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38111590 con 0x7f9e38102e80 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e3d774640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e38111890 con 0x7f9e3810b930 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34198/0 (socket says 172.21.15.73:34198) 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 -- 172.21.15.73:0/3065757348 learned_addr learned my addr 172.21.15.73:0/3065757348 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e377fe640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 0x7f9e38100c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e2ffff640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 0x7f9e381006f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 -- 172.21.15.73:0/3065757348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 msgr2=0x7f9e38100c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.956 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 0x7f9e38100c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 -- 172.21.15.73:0/3065757348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 msgr2=0x7f9e381006f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 0x7f9e381006f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.954+0000 7f9e36ffd640 1 -- 172.21.15.73:0/3065757348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e381a6090 con 0x7f9e38102560 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e36ffd640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006010 tx=0x7f9e2802cb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28041070 con 0x7f9e38102560 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e28004450 con 0x7f9e38102560 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e381a6320 con 0x7f9e38102560 2024-03-09T00:15:46.957 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e28003e00 con 0x7f9e38102560 2024-03-09T00:15:46.958 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.955+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e381a6860 con 0x7f9e38102560 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.956+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e380680e0 con 0x7f9e38102560 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.957+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9e28044070 con 0x7f9e38102560 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.958+0000 7f9e34ff9640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 0x7f9e240a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.958+0000 7f9e2ffff640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 0x7f9e240a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.958+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f9e280f0c50 con 0x7f9e38102560 2024-03-09T00:15:46.959 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.958+0000 7f9e2ffff640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 0x7f9e240a4ee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9e38101bf0 tx=0x7f9e20005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:46.960 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:46.962+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f9e280f6020 con 0x7f9e38102560 2024-03-09T00:15:47.175 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.177+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9e38106a70 con 0x7f9e38102560 2024-03-09T00:15:47.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.178+0000 7f9e34ff9640 1 -- 172.21.15.73:0/3065757348 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f9e38106a70 con 0x7f9e38102560 2024-03-09T00:15:47.176 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:15:47.176 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:15:47.177 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:15:47.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 msgr2=0x7f9e240a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 0x7f9e240a4ee0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f9e38101bf0 tx=0x7f9e20005f90 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 msgr2=0x7f9e381001b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 crypto rx=0x7f9e28006010 tx=0x7f9e2802cb30 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 shutdown_connections 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9e3810b930 0x7f9e38100c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9e38102e80 0x7f9e381006f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9e240a2a20 0x7f9e240a4ee0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 --2- 172.21.15.73:0/3065757348 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9e38102560 0x7f9e381001b0 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 >> 172.21.15.73:0/3065757348 conn(0x7f9e380772f0 msgr2=0x7f9e380fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.179 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 shutdown_connections 2024-03-09T00:15:47.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.180+0000 7f9e3d774640 1 -- 172.21.15.73:0/3065757348 wait complete. 2024-03-09T00:15:47.189 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:15:47.189 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:15:47.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.294+0000 7fcd4dd45640 1 -- 172.21.15.73:0/4290519054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd4810b850 msgr2=0x7fcd4810dcf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.294+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/4290519054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd4810b850 0x7fcd4810dcf0 secure :-1 s=READY pgs=948 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009ca0 tx=0x7fcd4002c180 comp rx=0 tx=0).stop 2024-03-09T00:15:47.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- 172.21.15.73:0/4290519054 shutdown_connections 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/4290519054 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd4810b850 0x7fcd4810dcf0 unknown :-1 s=CLOSED pgs=948 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/4290519054 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd48102e30 0x7fcd4810b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/4290519054 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 0x7fcd481028f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- 172.21.15.73:0/4290519054 >> 172.21.15.73:0/4290519054 conn(0x7fcd480773a0 msgr2=0x7fcd480777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- 172.21.15.73:0/4290519054 shutdown_connections 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- 172.21.15.73:0/4290519054 wait complete. 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 Processor -- start 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- start start 2024-03-09T00:15:47.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 0x7fcd4819d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 0x7fcd481978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48111610 con 0x7fcd48102e30 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48111490 con 0x7fcd4810b850 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.295+0000 7fcd4dd45640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd48111790 con 0x7fcd48102510 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34206/0 (socket says 172.21.15.73:34206) 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 -- 172.21.15.73:0/1820033917 learned_addr learned my addr 172.21.15.73:0/1820033917 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:47.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd47fff640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 0x7fcd481978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 0x7fcd4819d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 -- 172.21.15.73:0/1820033917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 msgr2=0x7fcd4819d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 0x7fcd4819d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 -- 172.21.15.73:0/1820033917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 msgr2=0x7fcd481978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 0x7fcd481978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 -- 172.21.15.73:0/1820033917 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd48198050 con 0x7fcd48102e30 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd47fff640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 0x7fcd481978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00e9f0 tx=0x7fcd3c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd44ff9640 1 -- 172.21.15.73:0/1820033917 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c00cd80 con 0x7fcd48102e30 2024-03-09T00:15:47.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd44ff9640 1 -- 172.21.15.73:0/1820033917 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd3c010910 con 0x7fcd48102e30 2024-03-09T00:15:47.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd48198340 con 0x7fcd48102e30 2024-03-09T00:15:47.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd44ff9640 1 -- 172.21.15.73:0/1820033917 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c018990 con 0x7fcd48102e30 2024-03-09T00:15:47.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.296+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd481a4590 con 0x7fcd48102e30 2024-03-09T00:15:47.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd44ff9640 1 -- 172.21.15.73:0/1820033917 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcd3c010ac0 con 0x7fcd48102e30 2024-03-09T00:15:47.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 Processor -- start 2024-03-09T00:15:47.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 -- start start 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea21160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea216a0 0x56297ea23a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297e9df360 con 0x56297ea181a0 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297e97bec0 con 0x56297ea18ac0 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.299+0000 7fcd50a02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297e924d30 con 0x56297ea216a0 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.309 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34222/0 (socket says 172.21.15.73:34222) 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 -- 172.21.15.73:0/911470603 learned_addr learned my addr 172.21.15.73:0/911470603 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd44ff9640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 0x7fcd300a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 0x7fcd300a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd477fe640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea21160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd44ff9640 1 -- 172.21.15.73:0/1820033917 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fcd3c0cf670 con 0x7fcd48102e30 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea216a0 0x56297ea23a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 -- 172.21.15.73:0/911470603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea216a0 msgr2=0x56297ea23a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.310 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea216a0 0x56297ea23a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 -- 172.21.15.73:0/911470603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 msgr2=0x56297ea21160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea21160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 -- 172.21.15.73:0/911470603 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56297e923bc0 con 0x56297ea181a0 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd477fe640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea21160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009ca0 tx=0x7fcd40003cc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=69808ff848d73a26 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd2dffb640 1 -- 172.21.15.73:0/911470603 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd40039040 con 0x56297ea181a0 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd2dffb640 1 -- 172.21.15.73:0/911470603 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd40004850 con 0x56297ea181a0 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd2dffb640 1 -- 172.21.15.73:0/911470603 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd400347c0 con 0x56297ea181a0 2024-03-09T00:15:47.311 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.300+0000 7fcd47fff640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 0x7fcd300a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcd4004c000 tx=0x7fcd40048000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- 172.21.15.73:0/911470603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 msgr2=0x56297ea18580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 crypto rx=0x7fcd40009ca0 tx=0x7fcd40003cc0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- 172.21.15.73:0/911470603 shutdown_connections 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea216a0 0x56297ea23a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea21160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 --2- 172.21.15.73:0/911470603 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea18580 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- 172.21.15.73:0/911470603 >> 172.21.15.73:0/911470603 conn(0x56297ea128d0 msgr2=0x56297ea12cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- 172.21.15.73:0/911470603 shutdown_connections 2024-03-09T00:15:47.312 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- 172.21.15.73:0/911470603 wait complete. 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 -- start start 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.301+0000 7fcd50a02740 1 Processor -- start 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1eb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1f070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297e91a4a0 con 0x56297ea181a0 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297e977f60 con 0x56297ea216a0 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x56297ea24490 con 0x56297ea18ac0 2024-03-09T00:15:47.313 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 10 client.0 _ll_register_callbacks cb 0x56297e975d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34224/0 (socket says 172.21.15.73:34224) 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 -- 172.21.15.73:0/1826796307 learned_addr learned my addr 172.21.15.73:0/1826796307 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd47fff640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 -- 172.21.15.73:0/1826796307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 msgr2=0x56297ea1eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.314 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 -- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 msgr2=0x56297ea1f070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1f070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 -- 172.21.15.73:0/1826796307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56297e6903f0 con 0x56297ea181a0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fcd380106d0 tx=0x7fcd38010b90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=188cfd0f3e406fd1 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3800eb30 con 0x56297ea181a0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fcd3800ecd0 con 0x56297ea181a0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.302+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd40004850 con 0x56297ea181a0 2024-03-09T00:15:47.315 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd3801c480 con 0x56297ea181a0 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 10 client.17058 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56297e75a7d0 con 0x56297ea181a0 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd40004850 con 0x56297ea181a0 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcd38011560 con 0x56297ea181a0 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 10 client.17058 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 10 client.17058 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 20 client.17058 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea31870 0x56297ea33c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.316 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 4 client.17058 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:47.317 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56297e989400 con 0x56297ea31870 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.303+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea31870 0x56297ea33c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.304+0000 7fcd26ffd640 10 client.17058 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.304+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea31870 0x56297ea33c80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9b4d8af192bf172 server_cookie=ab6add77bc3ae810 in_seq=0 out_seq=0 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.304+0000 7fcd26ffd640 1 -- 172.21.15.73:0/1826796307 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x56297e989400 con 0x56297ea31870 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.304+0000 7fcd26ffd640 10 client.17058 handle_command_reply: tid=0 2024-03-09T00:15:47.318 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.304+0000 7fcd50a02740 1 client.17058 shutdown 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 msgr2=0x56297ea1e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 crypto rx=0x7fcd380106d0 tx=0x7fcd38010b90 comp rx=0 tx=0).stop 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 shutdown_connections 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea31870 0x56297ea33c80 crc :-1 s=READY pgs=187 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1f070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd26ffd640 0 client.17058 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1826796307 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea1e5f0 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.319 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 >> 172.21.15.73:0/1826796307 conn(0x56297ea128d0 msgr2=0x56297ea14620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 shutdown_connections 2024-03-09T00:15:47.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 1 -- 172.21.15.73:0/1826796307 wait complete. 2024-03-09T00:15:47.320 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.305+0000 7fcd50a02740 20 client.17058 trim_cache size 0 max 16384 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 Processor -- start 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 -- start start 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1b850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481ac790 con 0x56297ea181a0 2024-03-09T00:15:47.326 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481aed30 con 0x56297ea216a0 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd50a02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481acd90 con 0x56297ea18ac0 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd477fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34236/0 (socket says 172.21.15.73:34236) 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.316+0000 7fcd477fe640 1 -- 172.21.15.73:0/1566822312 learned_addr learned my addr 172.21.15.73:0/1566822312 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 -- 172.21.15.73:0/1566822312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 msgr2=0x56297ea1d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.327 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 -- 172.21.15.73:0/1566822312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 msgr2=0x56297ea1b850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 -- 172.21.15.73:0/1566822312 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56297ea1cb40 con 0x56297ea181a0 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd477fe640 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcd38011900 tx=0x7fcd38003d40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2a04ceea9b70cdb server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd2d7fa640 1 -- 172.21.15.73:0/1566822312 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3802d070 con 0x56297ea181a0 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd2d7fa640 1 -- 172.21.15.73:0/1566822312 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd3800ee10 con 0x56297ea181a0 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd2d7fa640 1 -- 172.21.15.73:0/1566822312 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd38000e60 con 0x56297ea181a0 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd50a02740 1 -- 172.21.15.73:0/1566822312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 msgr2=0x56297ea20d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.317+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 crypto rx=0x7fcd38011900 tx=0x7fcd38003d40 comp rx=0 tx=0).stop 2024-03-09T00:15:47.328 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- 172.21.15.73:0/1566822312 shutdown_connections 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea216a0 0x56297ea1b850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea18ac0 0x56297ea1d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- 172.21.15.73:0/1566822312 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea181a0 0x56297ea20d20 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- 172.21.15.73:0/1566822312 >> 172.21.15.73:0/1566822312 conn(0x56297ea128d0 msgr2=0x56297ea1cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- 172.21.15.73:0/1566822312 shutdown_connections 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- 172.21.15.73:0/1566822312 wait complete. 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- start start 2024-03-09T00:15:47.329 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 Processor -- start 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea27810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea85df0 0x56297ea881e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481ad3b0 con 0x56297ea216a0 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481ae6d0 con 0x56297ea18ac0 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd481ada10 con 0x56297ea85df0 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd50a02740 10 client.0 _ll_register_callbacks cb 0x7fcd40037070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:47.330 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd46ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd46ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34246/0 (socket says 172.21.15.73:34246) 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd46ffd640 1 -- 172.21.15.73:0/794389072 learned_addr learned my addr 172.21.15.73:0/794389072 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.318+0000 7fcd47fff640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea85df0 0x56297ea881e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd477fe640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea27810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 -- 172.21.15.73:0/794389072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea85df0 msgr2=0x56297ea881e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea85df0 0x56297ea881e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 -- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 msgr2=0x56297ea27810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea27810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.331 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 -- 172.21.15.73:0/794389072 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56297ea887a0 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd46ffd640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c097d40 tx=0x7fcd3c000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b3a155617b7d946 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd3c097f50 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fcd3c0a3bf0 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcd3800ee10 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd3c0980f0 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 10 client.17064 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x56297ea89b00 con 0x56297ea216a0 2024-03-09T00:15:47.332 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd3800ee10 con 0x56297ea216a0 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fcd3c0cf290 con 0x56297ea216a0 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 10 client.17064 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 10 client.17064 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 20 client.17064 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea8a1e0 0x56297ea8c600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 4 client.17064 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.319+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56297e983750 con 0x56297ea8a1e0 2024-03-09T00:15:47.333 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd477fe640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea8a1e0 0x56297ea8c600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd477fe640 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea8a1e0 0x56297ea8c600 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15e633a920fac1f7 server_cookie=3f33580e0ae7a3d2 in_seq=0 out_seq=0 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd26ffd640 10 client.17064 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd26ffd640 1 -- 172.21.15.73:0/794389072 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56297e983750 con 0x56297ea8a1e0 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd26ffd640 10 client.17064 handle_command_reply: tid=0 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 client.17064 shutdown 2024-03-09T00:15:47.334 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 msgr2=0x56297ea27d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c097d40 tx=0x7fcd3c000bc0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 shutdown_connections 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x56297ea8a1e0 0x56297ea8c600 crc :-1 s=READY pgs=188 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x56297ea85df0 0x56297ea881e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x56297ea216a0 0x56297ea27d50 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd26ffd640 0 client.17064 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 --2- 172.21.15.73:0/794389072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x56297ea18ac0 0x56297ea27810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 >> 172.21.15.73:0/794389072 conn(0x56297ea128d0 msgr2=0x56297ea1d750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.335 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 shutdown_connections 2024-03-09T00:15:47.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.320+0000 7fcd50a02740 1 -- 172.21.15.73:0/794389072 wait complete. 2024-03-09T00:15:47.336 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd50a02740 20 client.17064 trim_cache size 0 max 16384 2024-03-09T00:15:47.336 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 msgr2=0x7fcd300a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 0x7fcd300a4e30 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fcd4004c000 tx=0x7fcd40048000 comp rx=0 tx=0).stop 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 msgr2=0x7fcd4819dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 crypto rx=0x7fcd3c00e9f0 tx=0x7fcd3c00eeb0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 shutdown_connections 2024-03-09T00:15:47.338 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcd4810b850 0x7fcd481978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcd300a2970 0x7fcd300a4e30 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcd48102e30 0x7fcd4819dc10 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 --2- 172.21.15.73:0/1820033917 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcd48102510 0x7fcd4819d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 >> 172.21.15.73:0/1820033917 conn(0x7fcd480773a0 msgr2=0x7fcd480fd1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 shutdown_connections 2024-03-09T00:15:47.339 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:47.321+0000 7fcd4dd45640 1 -- 172.21.15.73:0/1820033917 wait complete. 2024-03-09T00:15:48.341 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:15:48.341 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:15:48.341 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:15:48.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 -- 172.21.15.73:0/3572812529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641014f0 msgr2=0x7f72641018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 --2- 172.21.15.73:0/3572812529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641014f0 0x7f72641018d0 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 crypto rx=0x7f7254009900 tx=0x7f725402bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 -- 172.21.15.73:0/3572812529 shutdown_connections 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 --2- 172.21.15.73:0/3572812529 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f72641065c0 0x7f726410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 --2- 172.21.15.73:0/3572812529 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7264101ea0 0x7f7264105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 --2- 172.21.15.73:0/3572812529 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641014f0 0x7f72641018d0 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 -- 172.21.15.73:0/3572812529 >> 172.21.15.73:0/3572812529 conn(0x7f72640786f0 msgr2=0x7f7264078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 -- 172.21.15.73:0/3572812529 shutdown_connections 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.460+0000 7f7269e68640 1 -- 172.21.15.73:0/3572812529 wait complete. 2024-03-09T00:15:48.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 Processor -- start 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 -- start start 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72641014f0 0x7f726419d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7264197900 0x7f7264197db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264111680 con 0x7f72641065c0 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264111500 con 0x7f72641014f0 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7269e68640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7264111800 con 0x7f7264197900 2024-03-09T00:15:48.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34256/0 (socket says 172.21.15.73:34256) 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7262ffd640 1 -- 172.21.15.73:0/156023111 learned_addr learned my addr 172.21.15.73:0/156023111 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f72637fe640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72641014f0 0x7f726419d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.461+0000 7f7263fff640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7264197900 0x7f7264197db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 -- 172.21.15.73:0/156023111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7264197900 msgr2=0x7f7264197db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7264197900 0x7f7264197db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 -- 172.21.15.73:0/156023111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72641014f0 msgr2=0x7f726419d750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72641014f0 0x7f726419d750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 -- 172.21.15.73:0/156023111 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7264198420 con 0x7f72641065c0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7262ffd640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f725800c970 tx=0x7f725800ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7260ff9640 1 -- 172.21.15.73:0/156023111 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7258014070 con 0x7f72641065c0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7264198710 con 0x7f72641065c0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7260ff9640 1 -- 172.21.15.73:0/156023111 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7258004440 con 0x7f72641065c0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7260ff9640 1 -- 172.21.15.73:0/156023111 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725800f450 con 0x7f72641065c0 2024-03-09T00:15:48.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.462+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72641a4960 con 0x7f72641065c0 2024-03-09T00:15:48.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7260ff9640 1 -- 172.21.15.73:0/156023111 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7258017070 con 0x7f72641065c0 2024-03-09T00:15:48.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 Processor -- start 2024-03-09T00:15:48.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 -- start start 2024-03-09T00:15:48.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb20e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.472 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb2e150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5ea84b40 con 0x555d5eb2e690 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5ea2dd80 con 0x555d5eb20ab0 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f726ca49740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5ea80be0 con 0x555d5eb213d0 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7260ff9640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 0x7f724c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34262/0 (socket says 172.21.15.73:34262) 2024-03-09T00:15:48.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7262ffd640 1 -- 172.21.15.73:0/2216969072 learned_addr learned my addr 172.21.15.73:0/2216969072 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f72637fe640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb2e150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.465+0000 7f7263fff640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 0x7f724c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7263fff640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb20e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7260ff9640 1 -- 172.21.15.73:0/156023111 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f72580cf000 con 0x7f72641065c0 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 -- 172.21.15.73:0/2216969072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 msgr2=0x555d5eb2e150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb2e150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 -- 172.21.15.73:0/2216969072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 msgr2=0x555d5eb20e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb20e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 -- 172.21.15.73:0/2216969072 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d5e8617e0 con 0x555d5eb2e690 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7263fff640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb20e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7262ffd640 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f72580a34e0 tx=0x7f7258003920 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c3a75e7933bc9328 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7263fff640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 0x7f724c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f725c009fd0 tx=0x7f725c009840 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7249ffb640 1 -- 172.21.15.73:0/2216969072 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725809abd0 con 0x555d5eb2e690 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7249ffb640 1 -- 172.21.15.73:0/2216969072 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725809b970 con 0x555d5eb2e690 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.466+0000 7f7249ffb640 1 -- 172.21.15.73:0/2216969072 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725809b270 con 0x555d5eb2e690 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- 172.21.15.73:0/2216969072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 msgr2=0x555d5eb30a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 secure :-1 s=READY pgs=956 cs=0 l=1 rev1=1 crypto rx=0x7f72580a34e0 tx=0x7f7258003920 comp rx=0 tx=0).stop 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- 172.21.15.73:0/2216969072 shutdown_connections 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb30a80 unknown :-1 s=CLOSED pgs=956 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb2e150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 --2- 172.21.15.73:0/2216969072 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb20e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- 172.21.15.73:0/2216969072 >> 172.21.15.73:0/2216969072 conn(0x555d5eb1b4b0 msgr2=0x555d5eb1b890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- 172.21.15.73:0/2216969072 shutdown_connections 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- 172.21.15.73:0/2216969072 wait complete. 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 -- start start 2024-03-09T00:15:48.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.467+0000 7f726ca49740 1 Processor -- start 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb2e0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb2e690 0x555d5eb2bba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5eb31450 con 0x555d5eb213d0 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5ea411c0 con 0x555d5eb20ab0 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x555d5eb315d0 con 0x555d5eb2e690 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f726ca49740 10 client.0 _ll_register_callbacks cb 0x555d5ea307a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:48.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:34264/0 (socket says 172.21.15.73:34264) 2024-03-09T00:15:48.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 -- 172.21.15.73:0/3133005041 learned_addr learned my addr 172.21.15.73:0/3133005041 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f72637fe640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb2e0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7263fff640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb2e690 0x555d5eb2bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 -- 172.21.15.73:0/3133005041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb2e690 msgr2=0x555d5eb2bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb2e690 0x555d5eb2bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 -- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 msgr2=0x555d5eb2e0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb2e0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 -- 172.21.15.73:0/3133005041 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d5e8c1f70 con 0x555d5eb213d0 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7262ffd640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f72580cc410 tx=0x7f72580a3b00 comp rx=0 tx=0).ready entity=mon.0 client_cookie=21eabb35eba7240d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72580035f0 con 0x555d5eb213d0 2024-03-09T00:15:48.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.468+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f72580cc5e0 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f725809b970 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f725809b270 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 10 client.17076 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555d5e759170 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725809b970 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7258080080 con 0x555d5eb213d0 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 10 client.17076 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:48.480 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 10 client.17076 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 20 client.17076 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb3e680 0x555d5eb40a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 4 client.17076 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555d5ea5a960 con 0x555d5eb3e680 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.469+0000 7f72637fe640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb3e680 0x555d5eb40a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f7242ffd640 10 client.17076 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f72637fe640 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb3e680 0x555d5eb40a70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bc4f001d5be7ef60 server_cookie=e6a91f63b7f6bbff in_seq=0 out_seq=0 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f7242ffd640 1 -- 172.21.15.73:0/3133005041 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x555d5ea5a960 con 0x555d5eb3e680 2024-03-09T00:15:48.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f7242ffd640 10 client.17076 handle_command_reply: tid=0 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 client.17076 shutdown 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 msgr2=0x555d5eb2b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 crypto rx=0x7f72580cc410 tx=0x7f72580a3b00 comp rx=0 tx=0).stop 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 shutdown_connections 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb3e680 0x555d5eb40a70 crc :-1 s=READY pgs=189 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb2e690 0x555d5eb2bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f7242ffd640 0 client.17076 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb2b480 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 --2- 172.21.15.73:0/3133005041 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb2e0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.470+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 >> 172.21.15.73:0/3133005041 conn(0x555d5eb1b4b0 msgr2=0x555d5eb1d200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.471+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 shutdown_connections 2024-03-09T00:15:48.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.471+0000 7f726ca49740 1 -- 172.21.15.73:0/3133005041 wait complete. 2024-03-09T00:15:48.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.471+0000 7f726ca49740 20 client.17076 trim_cache size 0 max 16384 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 Processor -- start 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 -- start start 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb1d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb42a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641aa560 con 0x555d5eb2e690 2024-03-09T00:15:48.489 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641acb00 con 0x555d5eb20ab0 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f726ca49740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641aab60 con 0x555d5eb213d0 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49276/0 (socket says 172.21.15.73:49276) 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 -- 172.21.15.73:0/2218841616 learned_addr learned my addr 172.21.15.73:0/2218841616 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f72637fe640 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb1d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7262ffd640 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb42a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 -- 172.21.15.73:0/2218841616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 msgr2=0x555d5eb42a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.490 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb42a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 -- 172.21.15.73:0/2218841616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 msgr2=0x555d5eb1d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb1d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 -- 172.21.15.73:0/2218841616 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555d5eb21200 con 0x555d5eb2e690 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.486+0000 7f7263fff640 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f725c012740 tx=0x7f725c014040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d299e15b9ed53ed server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f72497fa640 1 -- 172.21.15.73:0/2218841616 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c02e070 con 0x555d5eb2e690 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f72497fa640 1 -- 172.21.15.73:0/2218841616 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c01fe50 con 0x555d5eb2e690 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f72497fa640 1 -- 172.21.15.73:0/2218841616 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725c029a50 con 0x555d5eb2e690 2024-03-09T00:15:48.491 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- 172.21.15.73:0/2218841616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 msgr2=0x555d5eb42fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 crypto rx=0x7f725c012740 tx=0x7f725c014040 comp rx=0 tx=0).stop 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- 172.21.15.73:0/2218841616 shutdown_connections 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb2e690 0x555d5eb42fd0 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb213d0 0x555d5eb42a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 --2- 172.21.15.73:0/2218841616 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb20ab0 0x555d5eb1d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- 172.21.15.73:0/2218841616 >> 172.21.15.73:0/2218841616 conn(0x555d5eb1b4b0 msgr2=0x555d5eb1c900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- 172.21.15.73:0/2218841616 shutdown_connections 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- 172.21.15.73:0/2218841616 wait complete. 2024-03-09T00:15:48.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.487+0000 7f726ca49740 1 -- start start 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 Processor -- start 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb20ab0 0x555d5eb41880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 0x555d5eb94760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ab180 con 0x555d5eb213d0 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ac4a0 con 0x555d5eb2e690 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72641ab7e0 con 0x555d5eb20ab0 2024-03-09T00:15:48.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f726ca49740 10 client.0 _ll_register_callbacks cb 0x7f725809b040 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49280/0 (socket says 172.21.15.73:49280) 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 -- 172.21.15.73:0/1291442930 learned_addr learned my addr 172.21.15.73:0/1291442930 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7263fff640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 0x555d5eb94760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f72637fe640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb20ab0 0x555d5eb41880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 -- 172.21.15.73:0/1291442930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb20ab0 msgr2=0x555d5eb41880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb20ab0 0x555d5eb41880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 -- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 msgr2=0x555d5eb94760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 0x555d5eb94760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7262ffd640 1 -- 172.21.15.73:0/1291442930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555d5eb94ca0 con 0x555d5eb213d0 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.488+0000 7f7263fff640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 0x555d5eb94760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7262ffd640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f72580976d0 tx=0x7f72580a3ac0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c382d57af712ff97 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f725809abd0 con 0x555d5eb213d0 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f72580978e0 con 0x555d5eb213d0 2024-03-09T00:15:48.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f725c01fe50 con 0x555d5eb213d0 2024-03-09T00:15:48.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f725809b970 con 0x555d5eb213d0 2024-03-09T00:15:48.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 10 client.17082 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:48.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x555d5eb95e60 con 0x555d5eb213d0 2024-03-09T00:15:48.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f725c01fe50 con 0x555d5eb213d0 2024-03-09T00:15:48.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f72580d21e0 con 0x555d5eb213d0 2024-03-09T00:15:48.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 10 client.17082 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:48.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 10 client.17082 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:48.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 20 client.17082 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb96320 0x555d5eb98740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 4 client.17082 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.489+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555d5ea46080 con 0x555d5eb96320 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.490+0000 7f72637fe640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb96320 0x555d5eb98740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.490+0000 7f7242ffd640 10 client.17082 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.490+0000 7f72637fe640 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb96320 0x555d5eb98740 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c30d6dc6411b247 server_cookie=e8cf00cf3943e9c6 in_seq=0 out_seq=0 2024-03-09T00:15:48.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.491+0000 7f7242ffd640 1 -- 172.21.15.73:0/1291442930 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555d5ea46080 con 0x555d5eb96320 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.491+0000 7f7242ffd640 10 client.17082 handle_command_reply: tid=0 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.491+0000 7f726ca49740 1 client.17082 shutdown 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.491+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 msgr2=0x555d5eb41dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.491+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 secure :-1 s=READY pgs=959 cs=0 l=1 rev1=1 crypto rx=0x7f72580976d0 tx=0x7f72580a3ac0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 shutdown_connections 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x555d5eb96320 0x555d5eb98740 crc :-1 s=READY pgs=190 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x555d5eb2e690 0x555d5eb94760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x555d5eb213d0 0x555d5eb41dc0 unknown :-1 s=CLOSED pgs=959 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7242ffd640 0 client.17082 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 --2- 172.21.15.73:0/1291442930 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x555d5eb20ab0 0x555d5eb41880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 >> 172.21.15.73:0/1291442930 conn(0x555d5eb1b4b0 msgr2=0x555d5eb3a0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 shutdown_connections 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 1 -- 172.21.15.73:0/1291442930 wait complete. 2024-03-09T00:15:48.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f726ca49740 20 client.17082 trim_cache size 0 max 16384 2024-03-09T00:15:48.502 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:48.502 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:15:48.502 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "451d3b30-c04c-40bc-83c2-b356a80096c0", 2024-03-09T00:15:48.502 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:15:48.502 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:48.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 msgr2=0x7f724c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 0x7f724c0a4e30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f725c009fd0 tx=0x7f725c009840 comp rx=0 tx=0).stop 2024-03-09T00:15:48.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 msgr2=0x7f726419dc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 crypto rx=0x7f725800c970 tx=0x7f725800ce30 comp rx=0 tx=0).stop 2024-03-09T00:15:48.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 shutdown_connections 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7264197900 0x7f7264197db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f724c0a2970 0x7f724c0a4e30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f72641065c0 0x7f726419dc90 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 --2- 172.21.15.73:0/156023111 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f72641014f0 0x7f726419d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 >> 172.21.15.73:0/156023111 conn(0x7f72640786f0 msgr2=0x7f72640ff2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 shutdown_connections 2024-03-09T00:15:48.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.492+0000 7f7269e68640 1 -- 172.21.15.73:0/156023111 wait complete. 2024-03-09T00:15:48.505 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:48 smithi073 ceph-mon[32309]: pgmap v277: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:48.505 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:48 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3065757348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:48.506 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:15:48.506 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:15:48.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:48 smithi102 ceph-mon[36106]: pgmap v277: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:48.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:48 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3065757348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:48.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:48 smithi179 ceph-mon[36637]: pgmap v277: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:15:48.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:48 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3065757348' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:15:48.614 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 -- 172.21.15.73:0/1778551397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c100d80 msgr2=0x7f601c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 --2- 172.21.15.73:0/1778551397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c100d80 0x7f601c104dc0 secure :-1 s=READY pgs=960 cs=0 l=1 rev1=1 crypto rx=0x7f600c009900 tx=0x7f600c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 -- 172.21.15.73:0/1778551397 shutdown_connections 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 --2- 172.21.15.73:0/1778551397 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 0x7f601c10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 --2- 172.21.15.73:0/1778551397 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c100d80 0x7f601c104dc0 unknown :-1 s=CLOSED pgs=960 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 --2- 172.21.15.73:0/1778551397 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1003d0 0x7f601c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 -- 172.21.15.73:0/1778551397 >> 172.21.15.73:0/1778551397 conn(0x7f601c0772f0 msgr2=0x7f601c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 -- 172.21.15.73:0/1778551397 shutdown_connections 2024-03-09T00:15:48.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.616+0000 7f6023ed7640 1 -- 172.21.15.73:0/1778551397 wait complete. 2024-03-09T00:15:48.617 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 Processor -- start 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 -- start start 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 0x7f601c198f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1994c0 0x7f601c1a3a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c110510 con 0x7f601c1003d0 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c110390 con 0x7f601c1054a0 2024-03-09T00:15:48.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6023ed7640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c110690 con 0x7f601c1994c0 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49286/0 (socket says 172.21.15.73:49286) 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 -- 172.21.15.73:0/174749767 learned_addr learned my addr 172.21.15.73:0/174749767 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6022474640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1994c0 0x7f601c1a3a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021472640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 0x7f601c198f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 -- 172.21.15.73:0/174749767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1994c0 msgr2=0x7f601c1a3a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1994c0 0x7f601c1a3a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 -- 172.21.15.73:0/174749767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 msgr2=0x7f601c198f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 0x7f601c198f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 -- 172.21.15.73:0/174749767 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f601c1a4270 con 0x7f601c1003d0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.617+0000 7f6021c73640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f601800d950 tx=0x7f601800dd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.618+0000 7f6012ffd640 1 -- 172.21.15.73:0/174749767 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018017070 con 0x7f601c1003d0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.618+0000 7f6012ffd640 1 -- 172.21.15.73:0/174749767 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f601800de80 con 0x7f601c1003d0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.618+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601c1a4560 con 0x7f601c1003d0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.618+0000 7f6012ffd640 1 -- 172.21.15.73:0/174749767 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601800f620 con 0x7f601c1003d0 2024-03-09T00:15:48.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.618+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c1a4a80 con 0x7f601c1003d0 2024-03-09T00:15:48.626 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.620+0000 7f6012ffd640 1 -- 172.21.15.73:0/174749767 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f6018009c30 con 0x7f601c1003d0 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 Processor -- start 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 -- start start 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6012ffd640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 0x7f60080a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 0x558eae9cdb40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6022474640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 0x7f60080a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.627 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 0x558eae9d6720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 0x558eae9d9050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae930e00 con 0x558eae9ce080 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae8d9b10 con 0x558eae9cd760 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6026b8a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae92cea0 con 0x558eae9d6c60 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6012ffd640 1 -- 172.21.15.73:0/174749767 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f60180cee50 con 0x7f601c1003d0 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6021472640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 0x558eae9d6720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6022474640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 0x558eae9d9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6021c73640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 0x558eae9cdb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6022474640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 0x558eae9d9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:43516/0 (socket says 172.21.15.73:43516) 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6021472640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 0x558eae9d6720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49288/0 (socket says 172.21.15.73:49288) 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.621+0000 7f6022474640 1 -- 172.21.15.73:0/3361202867 learned_addr learned my addr 172.21.15.73:0/3361202867 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 -- 172.21.15.73:0/3361202867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 msgr2=0x558eae9d9050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 0x558eae9d9050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 -- 172.21.15.73:0/3361202867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 msgr2=0x558eae9d6720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 0x558eae9d6720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 -- 172.21.15.73:0/3361202867 --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558eae70f7d0 con 0x558eae9cd760 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6022474640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 0x7f60080a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6014011810 tx=0x7f6014011c10 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6021c73640 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 0x558eae9cdb40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f60180cc1b0 tx=0x7f601809b900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f003fc666b533176 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6007fff640 1 -- 172.21.15.73:0/3361202867 <== mon.2 v2:172.21.15.102:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018020070 con 0x558eae9cd760 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6007fff640 1 -- 172.21.15.73:0/3361202867 <== mon.2 v2:172.21.15.102:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60180a3860 con 0x558eae9cd760 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.622+0000 7f6007fff640 1 -- 172.21.15.73:0/3361202867 <== mon.2 v2:172.21.15.102:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60180a33d0 con 0x558eae9cd760 2024-03-09T00:15:48.630 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- 172.21.15.73:0/3361202867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 msgr2=0x558eae9cdb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 0x558eae9cdb40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f60180cc1b0 tx=0x7f601809b900 comp rx=0 tx=0).stop 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- 172.21.15.73:0/3361202867 shutdown_connections 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9d6c60 0x558eae9d9050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9ce080 0x558eae9d6720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 --2- 172.21.15.73:0/3361202867 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9cd760 0x558eae9cdb40 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- 172.21.15.73:0/3361202867 >> 172.21.15.73:0/3361202867 conn(0x558eae9c7e90 msgr2=0x558eae9c8270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- 172.21.15.73:0/3361202867 shutdown_connections 2024-03-09T00:15:48.631 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- 172.21.15.73:0/3361202867 wait complete. 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.623+0000 7f6026b8a740 1 -- start start 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 Processor -- start 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9ce080 0x558eae9d3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9d6c60 0x558eae9d3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae9d9a50 con 0x558eae9cd760 2024-03-09T00:15:48.632 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae8ed0a0 con 0x558eae9d6c60 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x558eae9d9bd0 con 0x558eae9ce080 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6026b8a740 10 client.0 _ll_register_callbacks cb 0x558eae8eca70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021472640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9ce080 0x558eae9d3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49304/0 (socket says 172.21.15.73:49304) 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 -- 172.21.15.73:0/2705793172 learned_addr learned my addr 172.21.15.73:0/2705793172 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6022474640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9d6c60 0x558eae9d3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 -- 172.21.15.73:0/2705793172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9ce080 msgr2=0x558eae9d3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.633 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9ce080 0x558eae9d3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 -- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9d6c60 msgr2=0x558eae9d3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9d6c60 0x558eae9d3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.624+0000 7f6021c73640 1 -- 172.21.15.73:0/2705793172 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558eae6453f0 con 0x558eae9cd760 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6021c73640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f60180cc7b0 tx=0x7f6018000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4aabde53887451d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6018020050 con 0x558eae9cd760 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f60180a35e0 con 0x558eae9cd760 2024-03-09T00:15:48.634 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60180a3860 con 0x558eae9cd760 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6018097730 con 0x558eae9cd760 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 10 client.17094 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558eae6050f0 con 0x558eae9cd760 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60180a3860 con 0x558eae9cd760 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f60180b8020 con 0x558eae9cd760 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 10 client.17094 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 10 client.17094 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:48.635 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 20 client.17094 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.625+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eae9e64a0 0x558eae9e8890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.626+0000 7f6026b8a740 4 client.17094 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.626+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x558eae93e340 con 0x558eae9e64a0 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.626+0000 7f6021472640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eae9e64a0 0x558eae9e8890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.626+0000 7f6021472640 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eae9e64a0 0x558eae9e8890 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b0e232186107c736 server_cookie=e3d74d0e08ae2be4 in_seq=0 out_seq=0 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.626+0000 7f60057fa640 10 client.17094 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f60057fa640 1 -- 172.21.15.73:0/2705793172 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x558eae93e340 con 0x558eae9e64a0 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f60057fa640 10 client.17094 handle_command_reply: tid=0 2024-03-09T00:15:48.636 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 client.17094 shutdown 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 msgr2=0x558eae9d62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 crypto rx=0x7f60180cc7b0 tx=0x7f6018000bc0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 shutdown_connections 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eae9e64a0 0x558eae9e8890 crc :-1 s=READY pgs=191 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f60057fa640 0 client.17094 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9d6c60 0x558eae9d3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9ce080 0x558eae9d3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.637 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 --2- 172.21.15.73:0/2705793172 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9cd760 0x558eae9d62c0 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 >> 172.21.15.73:0/2705793172 conn(0x558eae9c7e90 msgr2=0x558eae9c9be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.627+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 shutdown_connections 2024-03-09T00:15:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.628+0000 7f6026b8a740 1 -- 172.21.15.73:0/2705793172 wait complete. 2024-03-09T00:15:48.638 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.628+0000 7f6026b8a740 20 client.17094 trim_cache size 0 max 16384 2024-03-09T00:15:48.644 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 Processor -- start 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 -- start start 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9cecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9d5d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ac790 con 0x558eae9d6c60 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1aed30 con 0x558eae9ce080 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6026b8a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1acd90 con 0x558eae9cd760 2024-03-09T00:15:48.645 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49320/0 (socket says 172.21.15.73:49320) 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 -- 172.21.15.73:0/4222227174 learned_addr learned my addr 172.21.15.73:0/4222227174 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6021472640 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9d5d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6021c73640 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9cecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 -- 172.21.15.73:0/4222227174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 msgr2=0x558eae9cecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9cecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 -- 172.21.15.73:0/4222227174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 msgr2=0x558eae9d5d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.646 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9d5d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.645+0000 7f6022474640 1 -- 172.21.15.73:0/4222227174 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x558eae9ccb10 con 0x558eae9d6c60 2024-03-09T00:15:48.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.646+0000 7f6022474640 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f601400f790 tx=0x7f6014020000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=df5ca24e77d88bcb server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.646+0000 7f60077fe640 1 -- 172.21.15.73:0/4222227174 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601402d070 con 0x558eae9d6c60 2024-03-09T00:15:48.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.646+0000 7f60077fe640 1 -- 172.21.15.73:0/4222227174 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6014004540 con 0x558eae9d6c60 2024-03-09T00:15:48.647 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.646+0000 7f60077fe640 1 -- 172.21.15.73:0/4222227174 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6014011e60 con 0x558eae9d6c60 2024-03-09T00:15:48.651 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- 172.21.15.73:0/4222227174 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 msgr2=0x558eae9d62c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 secure :-1 s=READY pgs=963 cs=0 l=1 rev1=1 crypto rx=0x7f601400f790 tx=0x7f6014020000 comp rx=0 tx=0).stop 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- 172.21.15.73:0/4222227174 shutdown_connections 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9d62c0 unknown :-1 s=CLOSED pgs=963 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9d5d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- 172.21.15.73:0/4222227174 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9cecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- 172.21.15.73:0/4222227174 >> 172.21.15.73:0/4222227174 conn(0x558eae9c7e90 msgr2=0x558eae9ce5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.652 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- 172.21.15.73:0/4222227174 shutdown_connections 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- 172.21.15.73:0/4222227174 wait complete. 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- start start 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 Processor -- start 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9daab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9daff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ad3b0 con 0x558eae9d6c60 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ae6d0 con 0x558eae9ce080 2024-03-09T00:15:48.653 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1ada10 con 0x558eae9cd760 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.647+0000 7f6026b8a740 10 client.0 _ll_register_callbacks cb 0x7f601809aaf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:49326/0 (socket says 172.21.15.73:49326) 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6021472640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9daff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 -- 172.21.15.73:0/265495027 learned_addr learned my addr 172.21.15.73:0/265495027 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6021c73640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9daab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.654 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 -- 172.21.15.73:0/265495027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 msgr2=0x558eae9daab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9daab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 -- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 msgr2=0x558eae9daff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9daff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 -- 172.21.15.73:0/265495027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558eaea3c9d0 con 0x558eae9d6c60 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6021472640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6021c73640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6022474640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6014005c10 tx=0x7f6014005fa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=837725d878d10c7a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f60057fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:15:48.655 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f601402d030 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(100..100 src has 1..100) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f6014004c40 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6014004540 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6014040050 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6026b8a740 10 client.17100 fetch_fsmap learned FSMap version 20 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.648+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x558eaea3dc80 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6014004540 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f601401d030 con 0x558eae9d6c60 2024-03-09T00:15:48.656 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 10 client.17100 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 10 client.17100 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 20 client.17100 populate_metadata read hostname 'smithi073' 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eaea3e340 0x558eaea40730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 4 client.17100 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x558eae938690 con 0x558eaea3e340 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6021c73640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eaea3e340 0x558eaea40730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f6021c73640 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eaea3e340 0x558eaea40730 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=778ac06c8e94e7ba server_cookie=1e625ad21ce8592b in_seq=0 out_seq=0 2024-03-09T00:15:48.657 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.649+0000 7f60057fa640 10 client.17100 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f60057fa640 1 -- 172.21.15.73:0/265495027 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x558eae938690 con 0x558eaea3e340 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f60057fa640 10 client.17100 handle_command_reply: tid=0 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 client.17100 shutdown 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 msgr2=0x558eae9db550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 crypto rx=0x7f6014005c10 tx=0x7f6014005fa0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 shutdown_connections 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x558eaea3e340 0x558eaea40730 crc :-1 s=READY pgs=192 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x558eae9d6c60 0x558eae9db550 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.658 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x558eae9ce080 0x558eae9daff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 --2- 172.21.15.73:0/265495027 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x558eae9cd760 0x558eae9daab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f60057fa640 0 client.17100 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 >> 172.21.15.73:0/265495027 conn(0x558eae9c7e90 msgr2=0x558eae9d1df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 shutdown_connections 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 1 -- 172.21.15.73:0/265495027 wait complete. 2024-03-09T00:15:48.659 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.650+0000 7f6026b8a740 20 client.17100 trim_cache size 0 max 16384 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2495 inodes in the stack)", 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "451d3b30-c04c-40bc-83c2-b356a80096c0": { 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "451d3b30-c04c-40bc-83c2-b356a80096c0", 2024-03-09T00:15:48.660 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:15:48.661 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:15:48.661 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:15:48.661 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:15:48.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 msgr2=0x7f60080a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.662 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 0x7f60080a4e30 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6014011810 tx=0x7f6014011c10 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 msgr2=0x7f601c1989f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 secure :-1 s=READY pgs=961 cs=0 l=1 rev1=1 crypto rx=0x7f601800d950 tx=0x7f601800dd50 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 shutdown_connections 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f601c1994c0 0x7f601c1a3a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f601c1054a0 0x7f601c198f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f60080a2970 0x7f60080a4e30 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 --2- 172.21.15.73:0/174749767 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f601c1003d0 0x7f601c1989f0 unknown :-1 s=CLOSED pgs=961 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:15:48.663 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 >> 172.21.15.73:0/174749767 conn(0x7f601c0772f0 msgr2=0x7f601c0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:15:48.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 shutdown_connections 2024-03-09T00:15:48.664 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:15:48.651+0000 7f6023ed7640 1 -- 172.21.15.73:0/174749767 wait complete. 2024-03-09T00:15:48.664 INFO:tasks.cephfs.filesystem:scrub status for tag:451d3b30-c04c-40bc-83c2-b356a80096c0 - {'path': '/', 'tag': '451d3b30-c04c-40bc-83c2-b356a80096c0', 'options': 'recursive,force'} 2024-03-09T00:15:49.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:49 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:15:49.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:49 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:49 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:15:49.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:49 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:49 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:15:49.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:49 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:15:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:49 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:49 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:15:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:49 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:49 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:15:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:49 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:15:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:49 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:49 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:15:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:49 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:15:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:49 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:15:50.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:50 smithi102 ceph-mon[36106]: pgmap v278: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:50.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:50 smithi179 ceph-mon[36637]: pgmap v278: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:50.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:50 smithi073 ceph-mon[32309]: pgmap v278: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:52.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:52 smithi102 ceph-mon[36106]: pgmap v279: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:52.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:52 smithi179 ceph-mon[36637]: pgmap v279: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:52.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:52 smithi073 ceph-mon[32309]: pgmap v279: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:15:53.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:53 smithi102 ceph-mon[36106]: pgmap v280: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:15:53.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:53 smithi179 ceph-mon[36637]: pgmap v280: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:15:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:53 smithi073 ceph-mon[32309]: pgmap v280: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:15:56.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:56 smithi102 ceph-mon[36106]: pgmap v281: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:15:56.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:56 smithi179 ceph-mon[36637]: pgmap v281: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:15:56.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:56 smithi073 ceph-mon[32309]: pgmap v281: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:15:58.456 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:15:58 smithi073 ceph-mon[32309]: pgmap v282: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:15:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:15:58 smithi102 ceph-mon[36106]: pgmap v282: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:15:58.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:15:58 smithi179 ceph-mon[36637]: pgmap v282: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:16:00.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:00 smithi102 ceph-mon[36106]: pgmap v283: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:00.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:00 smithi179 ceph-mon[36637]: pgmap v283: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:00.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:00 smithi073 ceph-mon[32309]: pgmap v283: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:01.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:01 smithi102 ceph-mon[36106]: pgmap v284: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:01.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:01 smithi179 ceph-mon[36637]: pgmap v284: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:01.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:01 smithi073 ceph-mon[32309]: pgmap v284: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:03.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:02 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:16:03.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:02 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:16:03.408 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:02 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:16:04.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:03 smithi102 ceph-mon[36106]: pgmap v285: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 44 KiB/s wr, 306 op/s 2024-03-09T00:16:04.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:16:04.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:16:04.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:03 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:16:04.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:03 smithi179 ceph-mon[36637]: pgmap v285: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 44 KiB/s wr, 306 op/s 2024-03-09T00:16:04.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:16:04.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:16:04.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:03 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:16:04.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:03 smithi073 ceph-mon[32309]: pgmap v285: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 298 KiB/s rd, 44 KiB/s wr, 306 op/s 2024-03-09T00:16:04.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:16:04.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:16:04.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:03 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:16:06.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:06 smithi102 ceph-mon[36106]: pgmap v286: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:06.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:06 smithi179 ceph-mon[36637]: pgmap v286: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:06.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:06 smithi073 ceph-mon[32309]: pgmap v286: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:08.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:08 smithi102 ceph-mon[36106]: pgmap v287: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:16:08.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:08 smithi179 ceph-mon[36637]: pgmap v287: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:16:08.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:08 smithi073 ceph-mon[32309]: pgmap v287: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:16:10.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:10 smithi102 ceph-mon[36106]: pgmap v288: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:10.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:10 smithi102 ceph-mon[36106]: osdmap e101: 12 total, 12 up, 12 in 2024-03-09T00:16:10.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:10 smithi179 ceph-mon[36637]: pgmap v288: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:10.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:10 smithi179 ceph-mon[36637]: osdmap e101: 12 total, 12 up, 12 in 2024-03-09T00:16:10.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:10 smithi073 ceph-mon[32309]: pgmap v288: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 44 KiB/s wr, 18 op/s 2024-03-09T00:16:10.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:10 smithi073 ceph-mon[32309]: osdmap e101: 12 total, 12 up, 12 in 2024-03-09T00:16:11.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:11 smithi102 ceph-mon[36106]: pgmap v290: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:16:11.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:11 smithi179 ceph-mon[36637]: pgmap v290: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:16:11.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:11 smithi073 ceph-mon[32309]: pgmap v290: 129 pgs: 129 active+clean; 35 MiB data, 893 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:16:14.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:14 smithi102 ceph-mon[36106]: pgmap v291: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:14.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:14 smithi179 ceph-mon[36637]: pgmap v291: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:14.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:14 smithi073 ceph-mon[32309]: pgmap v291: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:16.262 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:15 smithi102 ceph-mon[36106]: osdmap e102: 12 total, 12 up, 12 in 2024-03-09T00:16:16.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:15 smithi102 ceph-mon[36106]: pgmap v293: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:16:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:15 smithi179 ceph-mon[36637]: osdmap e102: 12 total, 12 up, 12 in 2024-03-09T00:16:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:15 smithi179 ceph-mon[36637]: pgmap v293: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:16:16.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:15 smithi073 ceph-mon[32309]: osdmap e102: 12 total, 12 up, 12 in 2024-03-09T00:16:16.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:15 smithi073 ceph-mon[32309]: pgmap v293: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:16:18.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:18 smithi102 ceph-mon[36106]: pgmap v294: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:16:18.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:18 smithi179 ceph-mon[36637]: pgmap v294: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:16:18.654 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:18 smithi073 ceph-mon[32309]: pgmap v294: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 255 B/s wr, 2 op/s 2024-03-09T00:16:18.665 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:16:18.666 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:16:18.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 -- 172.21.15.73:0/681709644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638069110 msgr2=0x7f0638105f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 --2- 172.21.15.73:0/681709644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638069110 0x7f0638105f50 secure :-1 s=READY pgs=965 cs=0 l=1 rev1=1 crypto rx=0x7f0628009900 tx=0x7f062802bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 -- 172.21.15.73:0/681709644 shutdown_connections 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 --2- 172.21.15.73:0/681709644 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 0x7f063810e1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 --2- 172.21.15.73:0/681709644 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638069110 0x7f0638105f50 unknown :-1 s=CLOSED pgs=965 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 --2- 172.21.15.73:0/681709644 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638068760 0x7f0638068b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 -- 172.21.15.73:0/681709644 >> 172.21.15.73:0/681709644 conn(0x7f06380772f0 msgr2=0x7f0638077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 -- 172.21.15.73:0/681709644 shutdown_connections 2024-03-09T00:16:18.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.782+0000 7f063f56f640 1 -- 172.21.15.73:0/681709644 wait complete. 2024-03-09T00:16:18.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 Processor -- start 2024-03-09T00:16:18.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 -- start start 2024-03-09T00:16:18.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 0x7f063819dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638197940 0x7f0638197df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0638111640 con 0x7f0638068760 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381114c0 con 0x7f0638197940 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063f56f640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381117c0 con 0x7f0638106630 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41982/0 (socket says 172.21.15.73:41982) 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063d30b640 1 -- 172.21.15.73:0/3622527366 learned_addr learned my addr 172.21.15.73:0/3622527366 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.783+0000 7f063db0c640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638197940 0x7f0638197df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.793 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063cb0a640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 0x7f063819dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 -- 172.21.15.73:0/3622527366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 msgr2=0x7f063819dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 0x7f063819dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 -- 172.21.15.73:0/3622527366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638197940 msgr2=0x7f0638197df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638197940 0x7f0638197df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 -- 172.21.15.73:0/3622527366 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0638198460 con 0x7f0638068760 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063d30b640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f063400c920 tx=0x7f063400cde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f062e7fc640 1 -- 172.21.15.73:0/3622527366 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634007c70 con 0x7f0638068760 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0638198750 con 0x7f0638068760 2024-03-09T00:16:18.794 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f062e7fc640 1 -- 172.21.15.73:0/3622527366 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0634007e10 con 0x7f0638068760 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f062e7fc640 1 -- 172.21.15.73:0/3622527366 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0634005260 con 0x7f0638068760 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.784+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06381a4950 con 0x7f0638068760 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.786+0000 7f062e7fc640 1 -- 172.21.15.73:0/3622527366 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0634005400 con 0x7f0638068760 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 Processor -- start 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 -- start start 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875278760 0x561875278b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.795 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875281c60 0x561875284050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618751dbe00 con 0x561875279080 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x561875184b10 con 0x561875281c60 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f0642220740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618751d7ea0 con 0x561875278760 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41984/0 (socket says 172.21.15.73:41984) 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 -- 172.21.15.73:0/1827285705 learned_addr learned my addr 172.21.15.73:0/1827285705 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f062e7fc640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 0x7f06240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063cb0a640 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875281c60 0x561875284050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.796 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063db0c640 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875278760 0x561875278b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063db0c640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 0x7f06240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 -- 172.21.15.73:0/1827285705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875278760 msgr2=0x561875278b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875278760 0x561875278b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 -- 172.21.15.73:0/1827285705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875281c60 msgr2=0x561875284050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875281c60 0x561875284050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f062e7fc640 1 -- 172.21.15.73:0/3622527366 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f06340cf3b0 con 0x7f0638068760 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.787+0000 7f063d30b640 1 -- 172.21.15.73:0/1827285705 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561874fba7d0 con 0x561875279080 2024-03-09T00:16:18.797 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f063d30b640 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f06340cc6b0 tx=0x7f0634050e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bf65beb6e403976a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f063db0c640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 0x7f06240a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0630005fd0 tx=0x7f0630009600 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f06237fe640 1 -- 172.21.15.73:0/1827285705 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06340056f0 con 0x561875279080 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f06237fe640 1 -- 172.21.15.73:0/1827285705 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063409bd20 con 0x561875279080 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f06237fe640 1 -- 172.21.15.73:0/1827285705 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f063400fb30 con 0x561875279080 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f0642220740 1 -- 172.21.15.73:0/1827285705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 msgr2=0x561875281720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.788+0000 7f0642220740 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 crypto rx=0x7f06340cc6b0 tx=0x7f0634050e20 comp rx=0 tx=0).stop 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- 172.21.15.73:0/1827285705 shutdown_connections 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875281c60 0x561875284050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.798 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875279080 0x561875281720 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- 172.21.15.73:0/1827285705 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875278760 0x561875278b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- 172.21.15.73:0/1827285705 >> 172.21.15.73:0/1827285705 conn(0x561875272e90 msgr2=0x561875273270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- 172.21.15.73:0/1827285705 shutdown_connections 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- 172.21.15.73:0/1827285705 wait complete. 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- start start 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 Processor -- start 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875281400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.799 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x56187527e790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618752849f0 con 0x561875281c60 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5618751980a0 con 0x561875278760 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x561875284b70 con 0x561875279080 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.789+0000 7f0642220740 10 client.0 _ll_register_callbacks cb 0x561875197a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:41998/0 (socket says 172.21.15.73:41998) 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 -- 172.21.15.73:0/671840733 learned_addr learned my addr 172.21.15.73:0/671840733 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.800 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063d30b640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875281400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063cb0a640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x56187527e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 -- 172.21.15.73:0/671840733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 msgr2=0x56187527e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x56187527e790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 -- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 msgr2=0x561875281400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875281400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 -- 172.21.15.73:0/671840733 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561874ef03f0 con 0x561875281c60 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f063db0c640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0630010a10 tx=0x7f0630004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=88a7a82c4b87786f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.801 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0620ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630025070 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f0630004d20 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f063409bd20 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0630018070 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0642220740 10 client.17112 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x561874eb00f0 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.790+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063409bd20 con 0x561875281c60 2024-03-09T00:16:18.802 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f063000f040 con 0x561875281c60 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 10 client.17112 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 10 client.17112 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 20 client.17112 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752916a0 0x561875293b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 4 client.17112 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5618751e9340 con 0x5618752916a0 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f063d30b640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752916a0 0x561875293b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.803 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f0620ff9640 10 client.17112 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.791+0000 7f063d30b640 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752916a0 0x561875293b10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84722572b257e02 server_cookie=3a2f1e9d6e4a3102 in_seq=0 out_seq=0 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0620ff9640 1 -- 172.21.15.73:0/671840733 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5618751e9340 con 0x5618752916a0 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0620ff9640 10 client.17112 handle_command_reply: tid=0 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 client.17112 shutdown 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 msgr2=0x56187527ecd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 crypto rx=0x7f0630010a10 tx=0x7f0630004a40 comp rx=0 tx=0).stop 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 shutdown_connections 2024-03-09T00:16:18.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752916a0 0x561875293b10 crc :-1 s=READY pgs=193 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x56187527ecd0 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x56187527e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0620ff9640 0 client.17112 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 --2- 172.21.15.73:0/671840733 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875281400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 >> 172.21.15.73:0/671840733 conn(0x561875272e90 msgr2=0x561875274be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 shutdown_connections 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 1 -- 172.21.15.73:0/671840733 wait complete. 2024-03-09T00:16:18.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.792+0000 7f0642220740 20 client.17112 trim_cache size 0 max 16384 2024-03-09T00:16:18.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 Processor -- start 2024-03-09T00:16:18.811 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 -- start start 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875289190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x561875279dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ac7d0 con 0x561875281c60 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381aed70 con 0x561875278760 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f0642220740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381acdd0 con 0x561875279080 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f063db0c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f063db0c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42012/0 (socket says 172.21.15.73:42012) 2024-03-09T00:16:18.812 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.807+0000 7f063db0c640 1 -- 172.21.15.73:0/2091014330 learned_addr learned my addr 172.21.15.73:0/2091014330 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063d30b640 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875289190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 -- 172.21.15.73:0/2091014330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 msgr2=0x561875279dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x561875279dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 -- 172.21.15.73:0/2091014330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 msgr2=0x561875289190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875289190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 -- 172.21.15.73:0/2091014330 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56187526d330 con 0x561875281c60 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f063db0c640 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0630005d70 tx=0x7f06300216a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=493cd6534d43beb6 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.813 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f0622ffd640 1 -- 172.21.15.73:0/2091014330 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0630025070 con 0x561875281c60 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f0622ffd640 1 -- 172.21.15.73:0/2091014330 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063001a040 con 0x561875281c60 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.808+0000 7f0622ffd640 1 -- 172.21.15.73:0/2091014330 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06300346b0 con 0x561875281c60 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- 172.21.15.73:0/2091014330 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 msgr2=0x561875280ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 secure :-1 s=READY pgs=969 cs=0 l=1 rev1=1 crypto rx=0x7f0630005d70 tx=0x7f06300216a0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- 172.21.15.73:0/2091014330 shutdown_connections 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875281c60 0x561875280ec0 unknown :-1 s=CLOSED pgs=969 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875279080 0x561875279dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- 172.21.15.73:0/2091014330 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875278760 0x561875289190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.814 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- 172.21.15.73:0/2091014330 >> 172.21.15.73:0/2091014330 conn(0x561875272e90 msgr2=0x5618752796c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- 172.21.15.73:0/2091014330 shutdown_connections 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- 172.21.15.73:0/2091014330 wait complete. 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- start start 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 Processor -- start 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875279080 0x5618752e6110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875281c60 0x5618752dffe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.815 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ad3f0 con 0x561875278760 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ae710 con 0x561875279080 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06381ada50 con 0x561875281c60 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f0642220740 10 client.0 _ll_register_callbacks cb 0x7f06300106c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f063d30b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42026/0 (socket says 172.21.15.73:42026) 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.809+0000 7f063d30b640 1 -- 172.21.15.73:0/2129980367 learned_addr learned my addr 172.21.15.73:0/2129980367 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063cb0a640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875279080 0x5618752e6110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.816 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063db0c640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875281c60 0x5618752dffe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 -- 172.21.15.73:0/2129980367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875281c60 msgr2=0x5618752dffe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875281c60 0x5618752dffe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 -- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875279080 msgr2=0x5618752e6110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875279080 0x5618752e6110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 -- 172.21.15.73:0/2129980367 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5618752e06d0 con 0x561875278760 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f063d30b640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0634097a80 tx=0x7f0634086040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c353304887c4674d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:18.817 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06340cce60 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f063400fb30 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f063001a040 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f063409f770 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0642220740 10 client.17118 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5618752ed3e0 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f063001a040 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.810+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f063409c950 con 0x561875278760 2024-03-09T00:16:18.818 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 10 client.17118 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 10 client.17118 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 20 client.17118 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752ed6d0 0x5618752efac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 4 client.17118 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5618751e3690 con 0x5618752ed6d0 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f063cb0a640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752ed6d0 0x5618752efac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0620ff9640 10 client.17118 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f063cb0a640 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752ed6d0 0x5618752efac0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6bb45214e8222fcb server_cookie=d026fbaf1e12b396 in_seq=0 out_seq=0 2024-03-09T00:16:18.819 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0620ff9640 1 -- 172.21.15.73:0/2129980367 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5618751e3690 con 0x5618752ed6d0 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0620ff9640 10 client.17118 handle_command_reply: tid=0 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.811+0000 7f0642220740 1 client.17118 shutdown 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 msgr2=0x5618752e5bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 crypto rx=0x7f0634097a80 tx=0x7f0634086040 comp rx=0 tx=0).stop 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 shutdown_connections 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5618752ed6d0 0x5618752efac0 crc :-1 s=READY pgs=194 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561875281c60 0x5618752dffe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0620ff9640 0 client.17118 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:18.820 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561875279080 0x5618752e6110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 --2- 172.21.15.73:0/2129980367 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561875278760 0x5618752e5bd0 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 >> 172.21.15.73:0/2129980367 conn(0x561875272e90 msgr2=0x56187527d060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 shutdown_connections 2024-03-09T00:16:18.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 1 -- 172.21.15.73:0/2129980367 wait complete. 2024-03-09T00:16:18.821 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f0642220740 20 client.17118 trim_cache size 0 max 16384 2024-03-09T00:16:18.822 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:18.822 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:16:18.822 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:16:18.822 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:18.823 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 msgr2=0x7f06240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 0x7f06240a4e30 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0630005fd0 tx=0x7f0630009600 comp rx=0 tx=0).stop 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 msgr2=0x7f063819d740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 crypto rx=0x7f063400c920 tx=0x7f063400cde0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 shutdown_connections 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0638197940 0x7f0638197df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0638106630 0x7f063819dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f06240a2970 0x7f06240a4e30 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.824 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 --2- 172.21.15.73:0/3622527366 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f0638068760 0x7f063819d740 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 >> 172.21.15.73:0/3622527366 conn(0x7f06380772f0 msgr2=0x7f06380fe2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 shutdown_connections 2024-03-09T00:16:18.825 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.812+0000 7f063f56f640 1 -- 172.21.15.73:0/3622527366 wait complete. 2024-03-09T00:16:18.825 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:16:18.825 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:16:18.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 -- 172.21.15.73:0/2863343778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801065c0 msgr2=0x7fbd8010e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 --2- 172.21.15.73:0/2863343778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801065c0 0x7fbd8010e190 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 crypto rx=0x7fbd7c009ca0 tx=0x7fbd7c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 -- 172.21.15.73:0/2863343778 shutdown_connections 2024-03-09T00:16:18.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 --2- 172.21.15.73:0/2863343778 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801065c0 0x7fbd8010e190 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 --2- 172.21.15.73:0/2863343778 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd80101ea0 0x7fbd80105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 --2- 172.21.15.73:0/2863343778 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd801014f0 0x7fbd801018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 -- 172.21.15.73:0/2863343778 >> 172.21.15.73:0/2863343778 conn(0x7fbd800fcd90 msgr2=0x7fbd800ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 -- 172.21.15.73:0/2863343778 shutdown_connections 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.928+0000 7fbd87344640 1 -- 172.21.15.73:0/2863343778 wait complete. 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 Processor -- start 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 -- start start 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd801065c0 0x7fbd8019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 0x7fbd80197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd801116d0 con 0x7fbd801014f0 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80111550 con 0x7fbd80197a10 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd87344640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd80111850 con 0x7fbd801065c0 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42048/0 (socket says 172.21.15.73:42048) 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 -- 172.21.15.73:0/3477152162 learned_addr learned my addr 172.21.15.73:0/3477152162 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:18.931 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd858e1640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 0x7fbd80197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 -- 172.21.15.73:0/3477152162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd801065c0 msgr2=0x7fbd8019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd801065c0 0x7fbd8019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 -- 172.21.15.73:0/3477152162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 msgr2=0x7fbd80197ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 0x7fbd80197ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 -- 172.21.15.73:0/3477152162 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd80198530 con 0x7fbd801014f0 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd858e1640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 0x7fbd80197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd850e0640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800d950 tx=0x7fbd7800de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.932 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.929+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd78017030 con 0x7fbd801014f0 2024-03-09T00:16:18.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.930+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd78004020 con 0x7fbd801014f0 2024-03-09T00:16:18.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.930+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd780047a0 con 0x7fbd801014f0 2024-03-09T00:16:18.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.930+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd80198820 con 0x7fbd801014f0 2024-03-09T00:16:18.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.930+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd801a4a20 con 0x7fbd801014f0 2024-03-09T00:16:18.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.931+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd80068530 con 0x7fbd801014f0 2024-03-09T00:16:18.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.932+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fbd780041d0 con 0x7fbd801014f0 2024-03-09T00:16:18.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.932+0000 7fbd767fc640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 0x7fbd6c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:18.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.933+0000 7fbd848df640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 0x7fbd6c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:18.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.933+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fbd780cecd0 con 0x7fbd801014f0 2024-03-09T00:16:18.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.933+0000 7fbd848df640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 0x7fbd6c0a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd80100480 tx=0x7fbd70006c90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:18.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:18.937+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fbd7809b640 con 0x7fbd801014f0 2024-03-09T00:16:19.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.147+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd80198ef0 con 0x7fbd801014f0 2024-03-09T00:16:19.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.148+0000 7fbd767fc640 1 -- 172.21.15.73:0/3477152162 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fbd7809af40 con 0x7fbd801014f0 2024-03-09T00:16:19.146 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:16:19.147 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:16:19.147 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:16:19.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 msgr2=0x7fbd6c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 0x7fbd6c0a4e30 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fbd80100480 tx=0x7fbd70006c90 comp rx=0 tx=0).stop 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 msgr2=0x7fbd8019d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 secure :-1 s=READY pgs=972 cs=0 l=1 rev1=1 crypto rx=0x7fbd7800d950 tx=0x7fbd7800de10 comp rx=0 tx=0).stop 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 shutdown_connections 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fbd80197a10 0x7fbd80197ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fbd801065c0 0x7fbd8019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fbd6c0a2970 0x7fbd6c0a4e30 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 --2- 172.21.15.73:0/3477152162 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fbd801014f0 0x7fbd8019d810 unknown :-1 s=CLOSED pgs=972 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 >> 172.21.15.73:0/3477152162 conn(0x7fbd800fcd90 msgr2=0x7fbd800ff110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 shutdown_connections 2024-03-09T00:16:19.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.150+0000 7fbd87344640 1 -- 172.21.15.73:0/3477152162 wait complete. 2024-03-09T00:16:19.159 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:16:19.159 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:16:19.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 -- 172.21.15.73:0/906778038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010106310 msgr2=0x7f70101066f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 --2- 172.21.15.73:0/906778038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010106310 0x7f70101066f0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 crypto rx=0x7f7004009930 tx=0x7f700402bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 -- 172.21.15.73:0/906778038 shutdown_connections 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 --2- 172.21.15.73:0/906778038 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100e00 0x7f701010cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 --2- 172.21.15.73:0/906778038 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010100440 0x7f70101008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 --2- 172.21.15.73:0/906778038 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010106310 0x7f70101066f0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 -- 172.21.15.73:0/906778038 >> 172.21.15.73:0/906778038 conn(0x7f7010076f80 msgr2=0x7f7010077390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 -- 172.21.15.73:0/906778038 shutdown_connections 2024-03-09T00:16:19.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 -- 172.21.15.73:0/906778038 wait complete. 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.265+0000 7f701566c640 1 Processor -- start 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 -- start start 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 0x7f70101989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010106310 0x7f70101a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7010110520 con 0x7f7010100e00 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101103a0 con 0x7f7010106310 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f701566c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101106a0 con 0x7f7010100440 2024-03-09T00:16:19.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42060/0 (socket says 172.21.15.73:42060) 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 -- 172.21.15.73:0/2161049005 learned_addr learned my addr 172.21.15.73:0/2161049005 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700f7fe640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010106310 0x7f70101a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700effd640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 0x7f70101989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 -- 172.21.15.73:0/2161049005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 msgr2=0x7f70101989c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 0x7f70101989c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 -- 172.21.15.73:0/2161049005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010106310 msgr2=0x7f70101a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010106310 0x7f70101a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 -- 172.21.15.73:0/2161049005 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70101a3f80 con 0x7f7010100e00 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700effd640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 0x7f70101989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f700e7fc640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f700000d950 tx=0x7f700000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.266+0000 7f6ff7fff640 1 -- 172.21.15.73:0/2161049005 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7000017070 con 0x7f7010100e00 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.267+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70101a4150 con 0x7f7010100e00 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.267+0000 7f6ff7fff640 1 -- 172.21.15.73:0/2161049005 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7000009c30 con 0x7f7010100e00 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.267+0000 7f6ff7fff640 1 -- 172.21.15.73:0/2161049005 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700000f6b0 con 0x7f7010100e00 2024-03-09T00:16:19.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.267+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70101a4660 con 0x7f7010100e00 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f6ff7fff640 1 -- 172.21.15.73:0/2161049005 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f70000109d0 con 0x7f7010100e00 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 Processor -- start 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 -- start start 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 0x55989995c580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x559899967a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559899923360 con 0x55989995cac0 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598998bfec0 con 0x5598999656a0 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f701824e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559899868d30 con 0x55989995c1a0 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f6ff7fff640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 0x7f6ff80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42062/0 (socket says 172.21.15.73:42062) 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700effd640 1 -- 172.21.15.73:0/1854999119 learned_addr learned my addr 172.21.15.73:0/1854999119 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700f7fe640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 0x7f6ff80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f6ff7fff640 1 -- 172.21.15.73:0/2161049005 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f70000ceed0 con 0x7f7010100e00 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700effd640 1 -- 172.21.15.73:0/1854999119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 msgr2=0x55989995c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700f7fe640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 0x55989995c580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.269+0000 7f700e7fc640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x559899967a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700effd640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 0x55989995c580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700effd640 1 -- 172.21.15.73:0/1854999119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 msgr2=0x559899967a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700effd640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x559899967a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700effd640 1 -- 172.21.15.73:0/1854999119 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559899867bc0 con 0x55989995cac0 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700f7fe640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 0x55989995c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700e7fc640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x559899967a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700effd640 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7004006010 tx=0x7f700402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ffceb73eb9a370f server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f700f7fe640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 0x7f6ff80a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f70080098b0 tx=0x7f7008009840 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f6ff57fa640 1 -- 172.21.15.73:0/1854999119 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700402cb00 con 0x55989995cac0 2024-03-09T00:16:19.280 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f6ff57fa640 1 -- 172.21.15.73:0/1854999119 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700402cca0 con 0x55989995cac0 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f6ff57fa640 1 -- 172.21.15.73:0/1854999119 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f70040048c0 con 0x55989995cac0 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f701824e740 1 -- 172.21.15.73:0/1854999119 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 msgr2=0x559899965160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.270+0000 7f701824e740 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 secure :-1 s=READY pgs=975 cs=0 l=1 rev1=1 crypto rx=0x7f7004006010 tx=0x7f700402c650 comp rx=0 tx=0).stop 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- 172.21.15.73:0/1854999119 shutdown_connections 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x559899967a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995cac0 0x559899965160 unknown :-1 s=CLOSED pgs=975 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- 172.21.15.73:0/1854999119 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995c1a0 0x55989995c580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- 172.21.15.73:0/1854999119 >> 172.21.15.73:0/1854999119 conn(0x5598999568d0 msgr2=0x559899956cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- 172.21.15.73:0/1854999119 shutdown_connections 2024-03-09T00:16:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- 172.21.15.73:0/1854999119 wait complete. 2024-03-09T00:16:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- start start 2024-03-09T00:16:19.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 Processor -- start 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995c1a0 0x5598999625a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x559899962ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55989985e4a0 con 0x5598999656a0 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5598998bbf60 con 0x55989995c1a0 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559899968430 con 0x55989995cac0 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f701824e740 10 client.0 _ll_register_callbacks cb 0x5598998b9d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:19.283 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f700f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f700f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42074/0 (socket says 172.21.15.73:42074) 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f700f7fe640 1 -- 172.21.15.73:0/3686546134 learned_addr learned my addr 172.21.15.73:0/3686546134 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f700effd640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995c1a0 0x5598999625a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.271+0000 7f700e7fc640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x559899962ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 -- 172.21.15.73:0/3686546134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 msgr2=0x559899962ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x559899962ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.284 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 -- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995c1a0 msgr2=0x5598999625a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995c1a0 0x5598999625a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 -- 172.21.15.73:0/3686546134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5598995d43f0 con 0x5598999656a0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f700f7fe640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f70080109c0 tx=0x7f7008004a40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8b48e2e80f30a97a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f6ff2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700801c8f0 con 0x5598999656a0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f700801ca90 con 0x5598999656a0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f700402cca0 con 0x5598999656a0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7008031050 con 0x5598999656a0 2024-03-09T00:16:19.285 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f701824e740 10 client.17136 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55989969e7d0 con 0x5598999656a0 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.272+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700402cca0 con 0x5598999656a0 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f700800f040 con 0x5598999656a0 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 10 client.17136 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 10 client.17136 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 20 client.17136 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:19.286 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559899975730 0x559899977b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 4 client.17136 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5598998cd400 con 0x559899975730 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f700effd640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559899975730 0x559899977b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f6ff2ff5640 10 client.17136 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.273+0000 7f700effd640 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559899975730 0x559899977b20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4e80bae891072b2 server_cookie=6d1c3929fb6abc4e in_seq=0 out_seq=0 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/3686546134 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5598998cd400 con 0x559899975730 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f6ff2ff5640 10 client.17136 handle_command_reply: tid=0 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 client.17136 shutdown 2024-03-09T00:16:19.287 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 msgr2=0x559899963020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 secure :-1 s=READY pgs=976 cs=0 l=1 rev1=1 crypto rx=0x7f70080109c0 tx=0x7f7008004a40 comp rx=0 tx=0).stop 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 shutdown_connections 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559899975730 0x559899977b20 crc :-1 s=READY pgs=195 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5598999656a0 0x559899963020 unknown :-1 s=CLOSED pgs=976 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f6ff2ff5640 0 client.17136 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x559899962ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.288 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 --2- 172.21.15.73:0/3686546134 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995c1a0 0x5598999625a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 >> 172.21.15.73:0/3686546134 conn(0x5598999568d0 msgr2=0x559899958620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 shutdown_connections 2024-03-09T00:16:19.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 1 -- 172.21.15.73:0/3686546134 wait complete. 2024-03-09T00:16:19.289 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.274+0000 7f701824e740 20 client.17136 trim_cache size 0 max 16384 2024-03-09T00:16:19.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 Processor -- start 2024-03-09T00:16:19.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 -- start start 2024-03-09T00:16:19.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995cac0 0x55989996c660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5598999656a0 0x55989996cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101ac850 con 0x55989995c1a0 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101aedf0 con 0x55989995cac0 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f701824e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101ace50 con 0x5598999656a0 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42080/0 (socket says 172.21.15.73:42080) 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 -- 172.21.15.73:0/168978802 learned_addr learned my addr 172.21.15.73:0/168978802 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 -- 172.21.15.73:0/168978802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5598999656a0 msgr2=0x55989996cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5598999656a0 0x55989996cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 -- 172.21.15.73:0/168978802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995cac0 msgr2=0x55989996c660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995cac0 0x55989996c660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 -- 172.21.15.73:0/168978802 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559899956700 con 0x55989995c1a0 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f700effd640 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7004005c40 tx=0x7f700402bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=230a97188c61ddd3 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f6ff4ff9640 1 -- 172.21.15.73:0/168978802 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700402c2b0 con 0x55989995c1a0 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f6ff4ff9640 1 -- 172.21.15.73:0/168978802 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700402cc60 con 0x55989995c1a0 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.286+0000 7f6ff4ff9640 1 -- 172.21.15.73:0/168978802 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700404d6d0 con 0x55989995c1a0 2024-03-09T00:16:19.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- 172.21.15.73:0/168978802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 msgr2=0x559899964cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 crypto rx=0x7f7004005c40 tx=0x7f700402bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- 172.21.15.73:0/168978802 shutdown_connections 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5598999656a0 0x55989996cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55989995cac0 0x55989996c660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 --2- 172.21.15.73:0/168978802 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x559899964cd0 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- 172.21.15.73:0/168978802 >> 172.21.15.73:0/168978802 conn(0x5598999568d0 msgr2=0x55989995d8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- 172.21.15.73:0/168978802 shutdown_connections 2024-03-09T00:16:19.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- 172.21.15.73:0/168978802 wait complete. 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 -- start start 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.287+0000 7f701824e740 1 Processor -- start 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x55989996c850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x55989996aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101ad470 con 0x55989995c1a0 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101ae790 con 0x5598999656a0 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70101adad0 con 0x55989995cac0 2024-03-09T00:16:19.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 10 client.0 _ll_register_callbacks cb 0x7f7004009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42086/0 (socket says 172.21.15.73:42086) 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 -- 172.21.15.73:0/706291970 learned_addr learned my addr 172.21.15.73:0/706291970 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700e7fc640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x55989996c850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700f7fe640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x55989996aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 -- 172.21.15.73:0/706291970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 msgr2=0x55989996c850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x55989996c850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 -- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 msgr2=0x55989996aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:19.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x55989996aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 -- 172.21.15.73:0/706291970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55989996b170 con 0x55989995c1a0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f700effd640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f700402cad0 tx=0x7f700402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8136f45978ee7ccf server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f6ff2ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f700404dbe0 con 0x55989995c1a0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f7004052040 con 0x55989995c1a0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f700402cc60 con 0x55989995c1a0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f700404dd80 con 0x55989995c1a0 2024-03-09T00:16:19.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 10 client.17142 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.288+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5598999cc7a0 con 0x55989995c1a0 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f700402cc60 con 0x55989995c1a0 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7004003950 con 0x55989995c1a0 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 10 client.17142 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 10 client.17142 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 20 client.17142 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5598999cca70 0x5598999cee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 4 client.17142 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:16:19.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5598998c7750 con 0x5598999cca70 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f700e7fc640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5598999cca70 0x5598999cee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f6ff2ff5640 10 client.17142 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.289+0000 7f700e7fc640 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5598999cca70 0x5598999cee60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e8587d2cf1a5f679 server_cookie=d2d9b7ad5bf0496c in_seq=0 out_seq=0 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f6ff2ff5640 1 -- 172.21.15.73:0/706291970 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5598998c7750 con 0x5598999cca70 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f6ff2ff5640 10 client.17142 handle_command_reply: tid=0 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 client.17142 shutdown 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 msgr2=0x55989996c310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 crypto rx=0x7f700402cad0 tx=0x7f700402c650 comp rx=0 tx=0).stop 2024-03-09T00:16:19.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 shutdown_connections 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5598999cca70 0x5598999cee60 crc :-1 s=READY pgs=196 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5598999656a0 0x55989996aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f6ff2ff5640 0 client.17142 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55989995cac0 0x55989996c850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 --2- 172.21.15.73:0/706291970 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55989995c1a0 0x55989996c310 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 >> 172.21.15.73:0/706291970 conn(0x5598999568d0 msgr2=0x559899962560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 shutdown_connections 2024-03-09T00:16:19.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 1 -- 172.21.15.73:0/706291970 wait complete. 2024-03-09T00:16:19.305 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701824e740 20 client.17142 trim_cache size 0 max 16384 2024-03-09T00:16:19.305 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:16:19.306 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 msgr2=0x7f6ff80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 0x7f6ff80a4e30 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f70080098b0 tx=0x7f7008009840 comp rx=0 tx=0).stop 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 msgr2=0x7f7010198f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 crypto rx=0x7f700000d950 tx=0x7f700000de10 comp rx=0 tx=0).stop 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 shutdown_connections 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7010106310 0x7f70101a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f6ff80a2970 0x7f6ff80a4e30 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7010100e00 0x7f7010198f00 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.307 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 --2- 172.21.15.73:0/2161049005 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7010100440 0x7f70101989c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:19.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 >> 172.21.15.73:0/2161049005 conn(0x7f7010076f80 msgr2=0x7f70100ff1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:19.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 shutdown_connections 2024-03-09T00:16:19.308 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:19.290+0000 7f701566c640 1 -- 172.21.15.73:0/2161049005 wait complete. 2024-03-09T00:16:19.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:19 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3477152162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:19.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:19 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3477152162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:19.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:19 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3477152162' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:20.308 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:16:20.309 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:16:20.309 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:16:20.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 -- 172.21.15.73:0/4017918338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841065c0 msgr2=0x7f2d8410e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 --2- 172.21.15.73:0/4017918338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841065c0 0x7f2d8410e190 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009900 tx=0x7f2d7c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:20.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 -- 172.21.15.73:0/4017918338 shutdown_connections 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 --2- 172.21.15.73:0/4017918338 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841065c0 0x7f2d8410e190 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 --2- 172.21.15.73:0/4017918338 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d84101ea0 0x7f2d84105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 --2- 172.21.15.73:0/4017918338 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841014f0 0x7f2d841018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 -- 172.21.15.73:0/4017918338 >> 172.21.15.73:0/4017918338 conn(0x7f2d840786f0 msgr2=0x7f2d84078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 -- 172.21.15.73:0/4017918338 shutdown_connections 2024-03-09T00:16:20.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.427+0000 7f2d8968c640 1 -- 172.21.15.73:0/4017918338 wait complete. 2024-03-09T00:16:20.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 Processor -- start 2024-03-09T00:16:20.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 -- start start 2024-03-09T00:16:20.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 0x7f2d8419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 0x7f2d8419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d84111660 con 0x7f2d841978c0 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841114e0 con 0x7f2d841014f0 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.428+0000 7f2d8968c640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841117e0 con 0x7f2d841065c0 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42110/0 (socket says 172.21.15.73:42110) 2024-03-09T00:16:20.430 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 -- 172.21.15.73:0/2680354371 learned_addr learned my addr 172.21.15.73:0/2680354371 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 0x7f2d8419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 0x7f2d8419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 -- 172.21.15.73:0/2680354371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 msgr2=0x7f2d8419dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 0x7f2d8419dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 -- 172.21.15.73:0/2680354371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 msgr2=0x7f2d8419d710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 0x7f2d8419d710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 -- 172.21.15.73:0/2680354371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d841983e0 con 0x7f2d841978c0 2024-03-09T00:16:20.431 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 0x7f2d8419dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 0x7f2d8419d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d837fe640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009230 tx=0x7f2d7c02cba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d6bfff640 1 -- 172.21.15.73:0/2680354371 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c041070 con 0x7f2d841978c0 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d84198670 con 0x7f2d841978c0 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d6bfff640 1 -- 172.21.15.73:0/2680354371 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c004450 con 0x7f2d841978c0 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d841a4870 con 0x7f2d841978c0 2024-03-09T00:16:20.432 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.429+0000 7f2d6bfff640 1 -- 172.21.15.73:0/2680354371 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c04b060 con 0x7f2d841978c0 2024-03-09T00:16:20.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.431+0000 7f2d6bfff640 1 -- 172.21.15.73:0/2680354371 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2d7c044070 con 0x7f2d841978c0 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 Processor -- start 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 -- start start 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d6bfff640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 0x7f2d6c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d145ed40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d145f280 0x5606d1461670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d13b9b40 con 0x5606d1455d80 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d1362d80 con 0x5606d14566a0 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d8c262740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d13b5be0 con 0x5606d145f280 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 0x7f2d6c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42126/0 (socket says 172.21.15.73:42126) 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d82ffd640 1 -- 172.21.15.73:0/3114571564 learned_addr learned my addr 172.21.15.73:0/3114571564 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d6bfff640 1 -- 172.21.15.73:0/2680354371 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f2d7c0f11b0 con 0x7f2d841978c0 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.432+0000 7f2d837fe640 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d145f280 0x5606d1461670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d827fc640 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d145ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 -- 172.21.15.73:0/3114571564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d145f280 msgr2=0x5606d1461670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d145f280 0x5606d1461670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 -- 172.21.15.73:0/3114571564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 msgr2=0x5606d145ed40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d145ed40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 -- 172.21.15.73:0/3114571564 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606d11967e0 con 0x5606d1455d80 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 0x7f2d6c0a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d840fe9c0 tx=0x7f2d78009680 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007f40 tx=0x7f2d7400e720 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a88a635408d76c5a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d697fa640 1 -- 172.21.15.73:0/3114571564 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74014070 con 0x5606d1455d80 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d697fa640 1 -- 172.21.15.73:0/3114571564 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d74004d20 con 0x5606d1455d80 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.433+0000 7f2d697fa640 1 -- 172.21.15.73:0/3114571564 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74005480 con 0x5606d1455d80 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- 172.21.15.73:0/3114571564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 msgr2=0x5606d1456160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 crypto rx=0x7f2d74007f40 tx=0x7f2d7400e720 comp rx=0 tx=0).stop 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- 172.21.15.73:0/3114571564 shutdown_connections 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d145f280 0x5606d1461670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d145ed40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- 172.21.15.73:0/3114571564 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d1456160 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- 172.21.15.73:0/3114571564 >> 172.21.15.73:0/3114571564 conn(0x5606d14504b0 msgr2=0x5606d1450890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- 172.21.15.73:0/3114571564 shutdown_connections 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- 172.21.15.73:0/3114571564 wait complete. 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- start start 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 Processor -- start 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d14566a0 0x5606d145c030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d145c640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d1462070 con 0x5606d1455d80 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d13761c0 con 0x5606d145f280 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.434+0000 7f2d8c262740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606d14621f0 con 0x5606d14566a0 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d8c262740 10 client.0 _ll_register_callbacks cb 0x5606d13657a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42142/0 (socket says 172.21.15.73:42142) 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 -- 172.21.15.73:0/2347816520 learned_addr learned my addr 172.21.15.73:0/2347816520 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d837fe640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d145c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d14566a0 0x5606d145c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 -- 172.21.15.73:0/2347816520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d14566a0 msgr2=0x5606d145c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d14566a0 0x5606d145c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 -- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 msgr2=0x5606d145c640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d145c640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 -- 172.21.15.73:0/2347816520 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606d11f6f70 con 0x5606d1455d80 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000f80 tx=0x7f2d74007cb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1dadd90a0a517ca2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:20.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d74027060 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f2d74010640 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d74004d20 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d7402e050 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d8c262740 10 client.17154 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.435+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5606d108e170 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d74004d20 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2d74004040 con 0x5606d1455d80 2024-03-09T00:16:20.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 10 client.17154 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 10 client.17154 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 20 client.17154 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d146f330 0x5606d1471720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 4 client.17154 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5606d138f960 con 0x5606d146f330 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d146f330 0x5606d1471720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d827fc640 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d146f330 0x5606d1471720 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=232fa0368ac0daa5 server_cookie=a1e3fe0f76ebd095 in_seq=0 out_seq=0 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.436+0000 7f2d66ff5640 10 client.17154 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d66ff5640 1 -- 172.21.15.73:0/2347816520 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x5606d138f960 con 0x5606d146f330 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d66ff5640 10 client.17154 handle_command_reply: tid=0 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 client.17154 shutdown 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 msgr2=0x5606d145eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 crypto rx=0x7f2d74000f80 tx=0x7f2d74007cb0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 shutdown_connections 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d146f330 0x5606d1471720 crc :-1 s=READY pgs=197 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d145c640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d14566a0 0x5606d145c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 --2- 172.21.15.73:0/2347816520 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d1455d80 0x5606d145eca0 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d66ff5640 0 client.17154 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 >> 172.21.15.73:0/2347816520 conn(0x5606d14504b0 msgr2=0x5606d1452200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 shutdown_connections 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 1 -- 172.21.15.73:0/2347816520 wait complete. 2024-03-09T00:16:20.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.437+0000 7f2d8c262740 20 client.17154 trim_cache size 0 max 16384 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 Processor -- start 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 -- start start 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d145e760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d1460670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841ac760 con 0x5606d14566a0 2024-03-09T00:16:20.457 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841aed00 con 0x5606d145f280 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.452+0000 7f2d8c262740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841acd60 con 0x5606d1455d80 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42152/0 (socket says 172.21.15.73:42152) 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 -- 172.21.15.73:0/1155799395 learned_addr learned my addr 172.21.15.73:0/1155799395 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d145e760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d837fe640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d1460670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 -- 172.21.15.73:0/1155799395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 msgr2=0x5606d145e760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.458 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d145e760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 -- 172.21.15.73:0/1155799395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 msgr2=0x5606d1460670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d1460670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 -- 172.21.15.73:0/1155799395 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606d145c9b0 con 0x5606d14566a0 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d837fe640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d1460670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d827fc640 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2d7801a000 tx=0x7f2d7801b000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a6a5bbc27aff85da server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d68ff9640 1 -- 172.21.15.73:0/1155799395 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7802d070 con 0x5606d14566a0 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d68ff9640 1 -- 172.21.15.73:0/1155799395 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d780282c0 con 0x5606d14566a0 2024-03-09T00:16:20.459 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.453+0000 7f2d68ff9640 1 -- 172.21.15.73:0/1155799395 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78021730 con 0x5606d14566a0 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- 172.21.15.73:0/1155799395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 msgr2=0x5606d145eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 crypto rx=0x7f2d7801a000 tx=0x7f2d7801b000 comp rx=0 tx=0).stop 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- 172.21.15.73:0/1155799395 shutdown_connections 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d145f280 0x5606d1460670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d14566a0 0x5606d145eca0 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- 172.21.15.73:0/1155799395 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d145e760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- 172.21.15.73:0/1155799395 >> 172.21.15.73:0/1155799395 conn(0x5606d14504b0 msgr2=0x5606d14572e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- 172.21.15.73:0/1155799395 shutdown_connections 2024-03-09T00:16:20.460 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- 172.21.15.73:0/1155799395 wait complete. 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- start start 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 Processor -- start 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d14bf950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d14bfe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841ad380 con 0x5606d145f280 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841ae6a0 con 0x5606d14566a0 2024-03-09T00:16:20.461 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841ad9e0 con 0x5606d1455d80 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.454+0000 7f2d8c262740 10 client.0 _ll_register_callbacks cb 0x7f2d74007700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42160/0 (socket says 172.21.15.73:42160) 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 -- 172.21.15.73:0/921120783 learned_addr learned my addr 172.21.15.73:0/921120783 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d14bf950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d827fc640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d14bfe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 -- 172.21.15.73:0/921120783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 msgr2=0x5606d14bf950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.462 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d14bf950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 -- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 msgr2=0x5606d14bfe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d14bfe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 -- 172.21.15.73:0/921120783 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606d1463e10 con 0x5606d145f280 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d827fc640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d14bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d837fe640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c005940 tx=0x7f2d7c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e250bd5abc3fc5f0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d66ff5640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c0c1680 con 0x5606d145f280 2024-03-09T00:16:20.463 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f2d7c003d30 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d780282c0 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d7c0c5cd0 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d8c262740 10 client.17160 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5606d14c7070 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.455+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d780282c0 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f2d7c0c5840 con 0x5606d145f280 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 10 client.17160 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 10 client.17160 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:20.464 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 20 client.17160 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d14c76f0 0x5606d14c9b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 4 client.17160 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5606d137b080 con 0x5606d14c76f0 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d14c76f0 0x5606d14c9b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d66ff5640 10 client.17160 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.456+0000 7f2d82ffd640 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d14c76f0 0x5606d14c9b10 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a05616f366a3ace5 server_cookie=3f49a18178eeb826 in_seq=0 out_seq=0 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d66ff5640 1 -- 172.21.15.73:0/921120783 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5606d137b080 con 0x5606d14c76f0 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d66ff5640 10 client.17160 handle_command_reply: tid=0 2024-03-09T00:16:20.465 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 client.17160 shutdown 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 msgr2=0x5606d1463720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 secure :-1 s=READY pgs=984 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c005940 tx=0x7f2d7c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 shutdown_connections 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5606d14c76f0 0x5606d14c9b10 crc :-1 s=READY pgs=198 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5606d145f280 0x5606d1463720 unknown :-1 s=CLOSED pgs=984 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5606d14566a0 0x5606d14bfe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 --2- 172.21.15.73:0/921120783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5606d1455d80 0x5606d14bf950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 >> 172.21.15.73:0/921120783 conn(0x5606d14504b0 msgr2=0x5606d1451830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d66ff5640 0 client.17160 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 shutdown_connections 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 1 -- 172.21.15.73:0/921120783 wait complete. 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8c262740 20 client.17160 trim_cache size 0 max 16384 2024-03-09T00:16:20.467 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:20.468 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:16:20.468 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "2ab52f8c-37d2-468d-b317-3fbb6797faa9", 2024-03-09T00:16:20.468 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:16:20.468 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:20.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 msgr2=0x7f2d6c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 0x7f2d6c0a4e30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f2d840fe9c0 tx=0x7f2d78009680 comp rx=0 tx=0).stop 2024-03-09T00:16:20.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 msgr2=0x7f2d84197d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.458+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c009230 tx=0x7f2d7c02cba0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.469 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 shutdown_connections 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f2d6c0a2970 0x7f2d6c0a4e30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f2d841978c0 0x7f2d84197d70 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f2d841065c0 0x7f2d8419dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 --2- 172.21.15.73:0/2680354371 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f2d841014f0 0x7f2d8419d710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 >> 172.21.15.73:0/2680354371 conn(0x7f2d840786f0 msgr2=0x7f2d840ff1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 shutdown_connections 2024-03-09T00:16:20.470 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.459+0000 7f2d8968c640 1 -- 172.21.15.73:0/2680354371 wait complete. 2024-03-09T00:16:20.471 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:20 smithi073 ceph-mon[32309]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 206 B/s wr, 2 op/s 2024-03-09T00:16:20.472 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:16:20.472 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:16:20.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:20 smithi102 ceph-mon[36106]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 206 B/s wr, 2 op/s 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 -- 172.21.15.73:0/1040306432 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c100d80 msgr2=0x7f9c7c104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 --2- 172.21.15.73:0/1040306432 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c100d80 0x7f9c7c104dc0 secure :-1 s=READY pgs=985 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c009900 tx=0x7f9c6c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 -- 172.21.15.73:0/1040306432 shutdown_connections 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 --2- 172.21.15.73:0/1040306432 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 0x7f9c7c10d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 --2- 172.21.15.73:0/1040306432 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c100d80 0x7f9c7c104dc0 unknown :-1 s=CLOSED pgs=985 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 --2- 172.21.15.73:0/1040306432 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 0x7f9c7c1007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 -- 172.21.15.73:0/1040306432 >> 172.21.15.73:0/1040306432 conn(0x7f9c7c0772f0 msgr2=0x7f9c7c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 -- 172.21.15.73:0/1040306432 shutdown_connections 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.581+0000 7f9c83d48640 1 -- 172.21.15.73:0/1040306432 wait complete. 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 Processor -- start 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 -- start start 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 0x7f9c7c1989b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 0x7f9c7c198ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c110510 con 0x7f9c7c199480 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c110390 con 0x7f9c7c1054a0 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c83d48640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c110690 con 0x7f9c7c1003d0 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c81ae4640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 0x7f9c7c1989b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42182/0 (socket says 172.21.15.73:42182) 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 -- 172.21.15.73:0/379830176 learned_addr learned my addr 172.21.15.73:0/379830176 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c812e3640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 0x7f9c7c198ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 -- 172.21.15.73:0/379830176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 msgr2=0x7f9c7c1989b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 0x7f9c7c1989b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 -- 172.21.15.73:0/379830176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 msgr2=0x7f9c7c198ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.582+0000 7f9c822e5640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 0x7f9c7c198ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c822e5640 1 -- 172.21.15.73:0/379830176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c7c1a4230 con 0x7f9c7c199480 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c822e5640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9c7400eb10 tx=0x7f9c7400c2d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c72ffd640 1 -- 172.21.15.73:0/379830176 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74004b20 con 0x7f9c7c199480 2024-03-09T00:16:20.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c72ffd640 1 -- 172.21.15.73:0/379830176 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c74004cc0 con 0x7f9c7c199480 2024-03-09T00:16:20.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7c1a4520 con 0x7f9c7c199480 2024-03-09T00:16:20.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c72ffd640 1 -- 172.21.15.73:0/379830176 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c74010760 con 0x7f9c7c199480 2024-03-09T00:16:20.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.583+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7c1a49f0 con 0x7f9c7c199480 2024-03-09T00:16:20.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:20 smithi179 ceph-mon[36637]: pgmap v295: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 206 B/s wr, 2 op/s 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.585+0000 7f9c72ffd640 1 -- 172.21.15.73:0/379830176 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9c74005020 con 0x7f9c7c199480 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 Processor -- start 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 -- start start 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3acb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3af460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a302e00 con 0x55fd7a39f410 2024-03-09T00:16:20.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a2abb10 con 0x55fd7a39fd30 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c869fd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a2feea0 con 0x55fd7a3ad070 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c72ffd640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 0x7f9c680a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c81ae4640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3acb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.586+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42190/0 (socket says 172.21.15.73:42190) 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 -- 172.21.15.73:0/3054720594 learned_addr learned my addr 172.21.15.73:0/3054720594 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c822e5640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 0x7f9c680a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3af460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c72ffd640 1 -- 172.21.15.73:0/379830176 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f9c740cf760 con 0x7f9c7c199480 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 -- 172.21.15.73:0/3054720594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 msgr2=0x55fd7a3af460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3af460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 -- 172.21.15.73:0/3054720594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 msgr2=0x55fd7a3acb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3acb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 -- 172.21.15.73:0/3054720594 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd7a0e17d0 con 0x55fd7a39f410 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02cfd0 tx=0x7f9c6c005dd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6ad69d47b486c300 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c822e5640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 0x7f9c680a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c74061aa0 tx=0x7f9c7406b4a0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c67fff640 1 -- 172.21.15.73:0/3054720594 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c004030 con 0x55fd7a39f410 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c67fff640 1 -- 172.21.15.73:0/3054720594 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c6c004ba0 con 0x55fd7a39f410 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.587+0000 7f9c67fff640 1 -- 172.21.15.73:0/3054720594 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c005240 con 0x55fd7a39f410 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 -- 172.21.15.73:0/3054720594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 msgr2=0x55fd7a39f7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 secure :-1 s=READY pgs=987 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02cfd0 tx=0x7f9c6c005dd0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 -- 172.21.15.73:0/3054720594 shutdown_connections 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3af460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3acb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3054720594 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a39f7f0 unknown :-1 s=CLOSED pgs=987 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 -- 172.21.15.73:0/3054720594 >> 172.21.15.73:0/3054720594 conn(0x55fd7a399e90 msgr2=0x55fd7a39a270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 -- 172.21.15.73:0/3054720594 shutdown_connections 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.588+0000 7f9c869fd740 1 -- 172.21.15.73:0/3054720594 wait complete. 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 -- start start 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 Processor -- start 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3a9ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3aa000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a3afe50 con 0x55fd7a39f410 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a2bf0a0 con 0x55fd7a39fd30 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fd7a3affd0 con 0x55fd7a3ad070 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c869fd740 10 client.0 _ll_register_callbacks cb 0x55fd7a2bea70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42200/0 (socket says 172.21.15.73:42200) 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c812e3640 1 -- 172.21.15.73:0/2338867533 learned_addr learned my addr 172.21.15.73:0/2338867533 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3a9ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.589+0000 7f9c812e3640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3aa000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 -- 172.21.15.73:0/2338867533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 msgr2=0x55fd7a3aa000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3aa000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 -- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 msgr2=0x55fd7a3a9ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3a9ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 -- 172.21.15.73:0/2338867533 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd7a0173f0 con 0x55fd7a39f410 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c812e3640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c036040 tx=0x7f9c6c004f60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d8bc90dd24a6a756 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c657fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c045070 con 0x55fd7a39f410 2024-03-09T00:16:20.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f9c6c004240 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c6c004ba0 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c6c005040 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c869fd740 10 client.17172 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.590+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fd79fd70f0 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c6c004ba0 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9c6c003b80 con 0x55fd7a39f410 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 10 client.17172 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 10 client.17172 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:20.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 20 client.17172 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a3bc970 0x55fd7a3bed60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 4 client.17172 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fd7a310340 con 0x55fd7a3bc970 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.591+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a3bc970 0x55fd7a3bed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.592+0000 7f9c657fa640 10 client.17172 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.592+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a3bc970 0x55fd7a3bed60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1d838aa5d7f965e9 server_cookie=373d5c2d5de8a99d in_seq=0 out_seq=0 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c657fa640 1 -- 172.21.15.73:0/2338867533 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55fd7a310340 con 0x55fd7a3bc970 2024-03-09T00:16:20.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c657fa640 10 client.17172 handle_command_reply: tid=0 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 client.17172 shutdown 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 msgr2=0x55fd7a3ac730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c036040 tx=0x7f9c6c004f60 comp rx=0 tx=0).stop 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 shutdown_connections 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a3bc970 0x55fd7a3bed60 crc :-1 s=READY pgs=199 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a3ad070 0x55fd7a3aa000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c657fa640 0 client.17172 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3a9ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 --2- 172.21.15.73:0/2338867533 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3ac730 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 >> 172.21.15.73:0/2338867533 conn(0x55fd7a399e90 msgr2=0x55fd7a39bbe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 shutdown_connections 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 1 -- 172.21.15.73:0/2338867533 wait complete. 2024-03-09T00:16:20.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.593+0000 7f9c869fd740 20 client.17172 trim_cache size 0 max 16384 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 Processor -- start 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 -- start start 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39fd30 0x55fd7a3ac1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a3ad070 0x55fd7a3ac730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1aa5f0 con 0x55fd7a39f410 2024-03-09T00:16:20.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1acb90 con 0x55fd7a3ad070 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c869fd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1aabf0 con 0x55fd7a39fd30 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42206/0 (socket says 172.21.15.73:42206) 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c812e3640 1 -- 172.21.15.73:0/3290608451 learned_addr learned my addr 172.21.15.73:0/3290608451 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.611+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39fd30 0x55fd7a3ac1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a3ad070 0x55fd7a3ac730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 -- 172.21.15.73:0/3290608451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39fd30 msgr2=0x55fd7a3ac1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39fd30 0x55fd7a3ac1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 -- 172.21.15.73:0/3290608451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a3ad070 msgr2=0x55fd7a3ac730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a3ad070 0x55fd7a3ac730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 -- 172.21.15.73:0/3290608451 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fd7a3afea0 con 0x55fd7a39f410 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02cfd0 tx=0x7f9c6c02bc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b0eebad54ea004e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c677fe640 1 -- 172.21.15.73:0/3290608451 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c045070 con 0x55fd7a39f410 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c677fe640 1 -- 172.21.15.73:0/3290608451 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c6c0049d0 con 0x55fd7a39f410 2024-03-09T00:16:20.613 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.612+0000 7f9c677fe640 1 -- 172.21.15.73:0/3290608451 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c0044e0 con 0x55fd7a39f410 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- 172.21.15.73:0/3290608451 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 msgr2=0x55fd7a3a8aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02cfd0 tx=0x7f9c6c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- 172.21.15.73:0/3290608451 shutdown_connections 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a3ad070 0x55fd7a3ac730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39fd30 0x55fd7a3ac1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3290608451 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a39f410 0x55fd7a3a8aa0 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.618 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- 172.21.15.73:0/3290608451 >> 172.21.15.73:0/3290608451 conn(0x55fd7a399e90 msgr2=0x55fd7a3a8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- 172.21.15.73:0/3290608451 shutdown_connections 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- 172.21.15.73:0/3290608451 wait complete. 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 -- start start 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.613+0000 7f9c869fd740 1 Processor -- start 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39f410 0x55fd7a3c10e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3bd490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1ab210 con 0x55fd7a3ad070 2024-03-09T00:16:20.619 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1ac530 con 0x55fd7a39fd30 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7c1ab870 con 0x55fd7a39f410 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c869fd740 10 client.0 _ll_register_callbacks cb 0x7f9c7800b770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42220/0 (socket says 172.21.15.73:42220) 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 -- 172.21.15.73:0/3320690383 learned_addr learned my addr 172.21.15.73:0/3320690383 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3bd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39f410 0x55fd7a3c10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 -- 172.21.15.73:0/3320690383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39f410 msgr2=0x55fd7a3c10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.620 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39f410 0x55fd7a3c10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 -- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 msgr2=0x55fd7a3bd490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3bd490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.614+0000 7f9c812e3640 1 -- 172.21.15.73:0/3320690383 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fd7a3be200 con 0x55fd7a3ad070 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c812e3640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02c170 tx=0x7f9c6c02cbe0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=886de17e2a28ffe8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9c6c045070 con 0x55fd7a3ad070 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f9c6c04e700 con 0x55fd7a3ad070 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9c6c0049d0 con 0x55fd7a3ad070 2024-03-09T00:16:20.621 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9c6c0055e0 con 0x55fd7a3ad070 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 10 client.17178 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55fd7a413a40 con 0x55fd7a3ad070 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9c6c04ee00 con 0x55fd7a3ad070 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f9c6c061450 con 0x55fd7a3ad070 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 10 client.17178 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 10 client.17178 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:20.622 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 20 client.17178 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a413ec0 0x55fd7a4162b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 4 client.17178 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.615+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x7f9c6c0049d0 con 0x55fd7a413ec0 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.616+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a413ec0 0x55fd7a4162b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.616+0000 7f9c81ae4640 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a413ec0 0x55fd7a4162b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fef9ced2890f99ac server_cookie=8bed2adb4f7ecf60 in_seq=0 out_seq=0 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.616+0000 7f9c657fa640 10 client.17178 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c657fa640 1 -- 172.21.15.73:0/3320690383 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x7f9c6c0049d0 con 0x55fd7a413ec0 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c657fa640 10 client.17178 handle_command_reply: tid=0 2024-03-09T00:16:20.623 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 client.17178 shutdown 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 msgr2=0x55fd7a3bd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 crypto rx=0x7f9c6c02c170 tx=0x7f9c6c02cbe0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 shutdown_connections 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55fd7a413ec0 0x55fd7a4162b0 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55fd7a3ad070 0x55fd7a3bd9d0 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55fd7a39fd30 0x55fd7a3bd490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 --2- 172.21.15.73:0/3320690383 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55fd7a39f410 0x55fd7a3c10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c657fa640 0 client.17178 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:20.624 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 >> 172.21.15.73:0/3320690383 conn(0x55fd7a399e90 msgr2=0x55fd7a39b2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 shutdown_connections 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 1 -- 172.21.15.73:0/3320690383 wait complete. 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.617+0000 7f9c869fd740 20 client.17178 trim_cache size 0 max 16384 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2453 inodes in the stack)", 2024-03-09T00:16:20.625 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: "2ab52f8c-37d2-468d-b317-3fbb6797faa9": { 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "2ab52f8c-37d2-468d-b317-3fbb6797faa9", 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:16:20.626 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 msgr2=0x7f9c680a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 0x7f9c680a4e30 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f9c74061aa0 tx=0x7f9c7406b4a0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 msgr2=0x7f9c7c1a3a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 secure :-1 s=READY pgs=986 cs=0 l=1 rev1=1 crypto rx=0x7f9c7400eb10 tx=0x7f9c7400c2d0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 shutdown_connections 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f9c680a2970 0x7f9c680a4e30 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.628 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9c7c199480 0x7f9c7c1a3a30 unknown :-1 s=CLOSED pgs=986 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9c7c1054a0 0x7f9c7c198ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 --2- 172.21.15.73:0/379830176 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9c7c1003d0 0x7f9c7c1989b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:20.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 >> 172.21.15.73:0/379830176 conn(0x7f9c7c0772f0 msgr2=0x7f9c7c0fe400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:20.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 shutdown_connections 2024-03-09T00:16:20.629 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:20.618+0000 7f9c83d48640 1 -- 172.21.15.73:0/379830176 wait complete. 2024-03-09T00:16:20.629 INFO:tasks.cephfs.filesystem:scrub status for tag:2ab52f8c-37d2-468d-b317-3fbb6797faa9 - {'path': '/', 'tag': '2ab52f8c-37d2-468d-b317-3fbb6797faa9', 'options': 'recursive,force'} 2024-03-09T00:16:21.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:21 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:16:21.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:21 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:21 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:16:21.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:21 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:21 smithi102 ceph-mon[36106]: pgmap v296: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:21.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:21 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:16:21.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:21 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:21 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:16:21.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:21 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:21 smithi179 ceph-mon[36637]: pgmap v296: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:21.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:21 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:16:21.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:21 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:21 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:16:21.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:21 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:21.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:21 smithi073 ceph-mon[32309]: pgmap v296: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:16:24.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:24 smithi102 ceph-mon[36106]: pgmap v297: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:16:24.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:24 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:16:24.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:24 smithi179 ceph-mon[36637]: pgmap v297: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:16:24.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:24 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:16:24.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:24 smithi073 ceph-mon[32309]: pgmap v297: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 345 KiB/s rd, 102 B/s wr, 346 op/s 2024-03-09T00:16:24.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:24 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:16:25.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:25 smithi102 ceph-mon[36106]: pgmap v298: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:16:25.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:25 smithi179 ceph-mon[36637]: pgmap v298: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:16:25.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:25 smithi073 ceph-mon[32309]: pgmap v298: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 340 KiB/s rd, 100 B/s wr, 340 op/s 2024-03-09T00:16:28.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:28 smithi102 ceph-mon[36106]: pgmap v299: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:16:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:28 smithi179 ceph-mon[36637]: pgmap v299: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:16:28.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:28 smithi073 ceph-mon[32309]: pgmap v299: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 85 B/s wr, 289 op/s 2024-03-09T00:16:29.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:29 smithi102 ceph-mon[36106]: pgmap v300: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:29.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:29 smithi179 ceph-mon[36637]: pgmap v300: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:29.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:29 smithi073 ceph-mon[32309]: pgmap v300: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 287 KiB/s rd, 288 op/s 2024-03-09T00:16:32.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:32 smithi102 ceph-mon[36106]: pgmap v301: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:16:32.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:32 smithi179 ceph-mon[36637]: pgmap v301: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:16:32.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:32 smithi073 ceph-mon[32309]: pgmap v301: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 288 op/s 2024-03-09T00:16:33.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:33 smithi102 ceph-mon[36106]: pgmap v302: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:16:33.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:33 smithi179 ceph-mon[36637]: pgmap v302: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:16:33.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:33 smithi073 ceph-mon[32309]: pgmap v302: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 288 KiB/s rd, 289 op/s 2024-03-09T00:16:36.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:36 smithi102 ceph-mon[36106]: pgmap v303: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:36.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:36 smithi179 ceph-mon[36637]: pgmap v303: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:36.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:36 smithi073 ceph-mon[32309]: pgmap v303: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:37.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:37 smithi102 ceph-mon[36106]: pgmap v304: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:37.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:37 smithi179 ceph-mon[36637]: pgmap v304: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:37.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:37 smithi073 ceph-mon[32309]: pgmap v304: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:40.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:40 smithi102 ceph-mon[36106]: pgmap v305: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:40.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:40 smithi179 ceph-mon[36637]: pgmap v305: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:40.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:40 smithi073 ceph-mon[32309]: pgmap v305: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:41.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:41 smithi102 ceph-mon[36106]: pgmap v306: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:41.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:41 smithi179 ceph-mon[36637]: pgmap v306: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:41.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:41 smithi073 ceph-mon[32309]: pgmap v306: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:43.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:43 smithi102 ceph-mon[36106]: pgmap v307: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:43.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:43 smithi179 ceph-mon[36637]: pgmap v307: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:43.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:43 smithi073 ceph-mon[32309]: pgmap v307: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:45.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:45 smithi102 ceph-mon[36106]: pgmap v308: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:45 smithi179 ceph-mon[36637]: pgmap v308: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:45.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:45 smithi073 ceph-mon[32309]: pgmap v308: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:47.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:47 smithi102 ceph-mon[36106]: pgmap v309: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:47.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:47 smithi179 ceph-mon[36637]: pgmap v309: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:47.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:47 smithi073 ceph-mon[32309]: pgmap v309: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:49.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:49 smithi102 ceph-mon[36106]: pgmap v310: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:49.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:49 smithi179 ceph-mon[36637]: pgmap v310: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:49 smithi073 ceph-mon[32309]: pgmap v310: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:50.631 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:16:50.631 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:16:50.748 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- 172.21.15.73:0/131860723 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4550100d80 msgr2=0x7f4550104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 --2- 172.21.15.73:0/131860723 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4550100d80 0x7f4550104dc0 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 crypto rx=0x7f4540009900 tx=0x7f454002bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:50.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- 172.21.15.73:0/131860723 shutdown_connections 2024-03-09T00:16:50.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 --2- 172.21.15.73:0/131860723 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f45501054a0 0x7f455010d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 --2- 172.21.15.73:0/131860723 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f4550100d80 0x7f4550104dc0 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.749 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 --2- 172.21.15.73:0/131860723 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 0x7f45501007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- 172.21.15.73:0/131860723 >> 172.21.15.73:0/131860723 conn(0x7f45500772f0 msgr2=0x7f4550077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- 172.21.15.73:0/131860723 shutdown_connections 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- 172.21.15.73:0/131860723 wait complete. 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 Processor -- start 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.748+0000 7f455804d640 1 -- start start 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 0x7f455019d870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4550197a20 0x7f4550197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550110590 con 0x7f45501054a0 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550110410 con 0x7f45501003d0 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550110710 con 0x7f4550197a20 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42518/0 (socket says 172.21.15.73:42518) 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 -- 172.21.15.73:0/2763461704 learned_addr learned my addr 172.21.15.73:0/2763461704 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f4555de9640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 0x7f455019d870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 -- 172.21.15.73:0/2763461704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4550197a20 msgr2=0x7f4550197ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4550197a20 0x7f4550197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 -- 172.21.15.73:0/2763461704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 msgr2=0x7f455019d870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 0x7f455019d870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 -- 172.21.15.73:0/2763461704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4550198540 con 0x7f45501054a0 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f45555e8640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f454002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f4546ffd640 1 -- 172.21.15.73:0/2763461704 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540041070 con 0x7f45501054a0 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f4546ffd640 1 -- 172.21.15.73:0/2763461704 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4540004450 con 0x7f45501054a0 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45501987d0 con 0x7f45501054a0 2024-03-09T00:16:50.752 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f4546ffd640 1 -- 172.21.15.73:0/2763461704 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4540034440 con 0x7f45501054a0 2024-03-09T00:16:50.753 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.749+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501a49c0 con 0x7f45501054a0 2024-03-09T00:16:50.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.751+0000 7f4546ffd640 1 -- 172.21.15.73:0/2763461704 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f4540044070 con 0x7f45501054a0 2024-03-09T00:16:50.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 Processor -- start 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 -- start start 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a9080 0x55ceb01b1720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b4050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb010be00 con 0x55ceb01a8760 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb00b4b10 con 0x55ceb01b1c60 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f455ac2a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb0107ea0 con 0x55ceb01a9080 2024-03-09T00:16:50.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f45565ea640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f4546ffd640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 0x7f453c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f45565ea640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42520/0 (socket says 172.21.15.73:42520) 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f45565ea640 1 -- 172.21.15.73:0/4111102015 learned_addr learned my addr 172.21.15.73:0/4111102015 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f4555de9640 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a9080 0x55ceb01b1720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.752+0000 7f45565ea640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 0x7f453c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f4546ffd640 1 -- 172.21.15.73:0/2763461704 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f45400f0e40 con 0x7f45501054a0 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45555e8640 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 -- 172.21.15.73:0/4111102015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a9080 msgr2=0x55ceb01b1720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a9080 0x55ceb01b1720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 -- 172.21.15.73:0/4111102015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 msgr2=0x55ceb01b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 -- 172.21.15.73:0/4111102015 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ceafeea7d0 con 0x55ceb01a8760 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f45480126c0 tx=0x7f4548012b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=56071f8f0336ab2d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f453bfff640 1 -- 172.21.15.73:0/4111102015 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4548012050 con 0x55ceb01a8760 2024-03-09T00:16:50.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f453bfff640 1 -- 172.21.15.73:0/4111102015 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4548013040 con 0x55ceb01a8760 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f453bfff640 1 -- 172.21.15.73:0/4111102015 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454801b070 con 0x55ceb01a8760 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.753+0000 7f45565ea640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 0x7f453c0a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f454801d000 tx=0x7f454801b000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- 172.21.15.73:0/4111102015 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 msgr2=0x55ceb01a8b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 crypto rx=0x7f45480126c0 tx=0x7f4548012b80 comp rx=0 tx=0).stop 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- 172.21.15.73:0/4111102015 shutdown_connections 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a9080 0x55ceb01b1720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 --2- 172.21.15.73:0/4111102015 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a8760 0x55ceb01a8b40 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- 172.21.15.73:0/4111102015 >> 172.21.15.73:0/4111102015 conn(0x55ceb01a2e90 msgr2=0x55ceb01a3270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- 172.21.15.73:0/4111102015 shutdown_connections 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- 172.21.15.73:0/4111102015 wait complete. 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 -- start start 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.754+0000 7f455ac2a740 1 Processor -- start 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01a8760 0x55ceb01b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01b1c60 0x55ceb01aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb01b4a50 con 0x55ceb01a9080 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb00c80a0 con 0x55ceb01a8760 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ceb01b4bd0 con 0x55ceb01b1c60 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 10 client.0 _ll_register_callbacks cb 0x55ceb00c7a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42528/0 (socket says 172.21.15.73:42528) 2024-03-09T00:16:50.765 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 -- 172.21.15.73:0/615566512 learned_addr learned my addr 172.21.15.73:0/615566512 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f4555de9640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01a8760 0x55ceb01b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45565ea640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01b1c60 0x55ceb01aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 -- 172.21.15.73:0/615566512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01b1c60 msgr2=0x55ceb01aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01b1c60 0x55ceb01aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 -- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01a8760 msgr2=0x55ceb01b13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01a8760 0x55ceb01b13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 -- 172.21.15.73:0/615566512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ceafe203f0 con 0x55ceb01a9080 2024-03-09T00:16:50.766 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45555e8640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f45400ee360 tx=0x7f45400bd980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=30cc15f80a3cf70a server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45397fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454004b040 con 0x55ceb01a9080 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f45400bd420 con 0x55ceb01a9080 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.755+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f4548013040 con 0x55ceb01a9080 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45400c5960 con 0x55ceb01a9080 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 10 client.17190 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ceafde00f0 con 0x55ceb01a9080 2024-03-09T00:16:50.767 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4548013040 con 0x55ceb01a9080 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45400f2040 con 0x55ceb01a9080 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 10 client.17190 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 10 client.17190 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 20 client.17190 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb01c1650 0x55ceb01c3a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 4 client.17190 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ceb0119340 con 0x55ceb01c1650 2024-03-09T00:16:50.768 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.756+0000 7f4555de9640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb01c1650 0x55ceb01c3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f45397fa640 10 client.17190 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f4555de9640 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb01c1650 0x55ceb01c3a40 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b84c0fcbb214bf34 server_cookie=339efec7c856b610 in_seq=0 out_seq=0 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f45397fa640 1 -- 172.21.15.73:0/615566512 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ceb0119340 con 0x55ceb01c1650 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f45397fa640 10 client.17190 handle_command_reply: tid=0 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 client.17190 shutdown 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 msgr2=0x55ceb01ae770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 crypto rx=0x7f45400ee360 tx=0x7f45400bd980 comp rx=0 tx=0).stop 2024-03-09T00:16:50.769 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 shutdown_connections 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb01c1650 0x55ceb01c3a40 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01b1c60 0x55ceb01aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f45397fa640 0 client.17190 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01ae770 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 --2- 172.21.15.73:0/615566512 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01a8760 0x55ceb01b13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.757+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 >> 172.21.15.73:0/615566512 conn(0x55ceb01a2e90 msgr2=0x55ceb01a4be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.758+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 shutdown_connections 2024-03-09T00:16:50.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.758+0000 7f455ac2a740 1 -- 172.21.15.73:0/615566512 wait complete. 2024-03-09T00:16:50.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.758+0000 7f455ac2a740 20 client.17190 trim_cache size 0 max 16384 2024-03-09T00:16:50.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.772+0000 7f455ac2a740 1 Processor -- start 2024-03-09T00:16:50.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.772+0000 7f455ac2a740 1 -- start start 2024-03-09T00:16:50.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.772+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b0ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.772+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f455ac2a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501ac8c0 con 0x55ceb01a9080 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f455ac2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501aee60 con 0x55ceb01b1c60 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f455ac2a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501acec0 con 0x55ceb01a8760 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42542/0 (socket says 172.21.15.73:42542) 2024-03-09T00:16:50.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 -- 172.21.15.73:0/1786244454 learned_addr learned my addr 172.21.15.73:0/1786244454 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45565ea640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f4555de9640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b0ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 -- 172.21.15.73:0/1786244454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 msgr2=0x55ceb01b0ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b0ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 -- 172.21.15.73:0/1786244454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 msgr2=0x55ceb01b8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 -- 172.21.15.73:0/1786244454 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ceb01c3db0 con 0x55ceb01a9080 2024-03-09T00:16:50.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f4555de9640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f45555e8640 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f45400bd150 tx=0x7f4540005b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b4ac7649fa64d5e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f453b7fe640 1 -- 172.21.15.73:0/1786244454 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454004b070 con 0x55ceb01a9080 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f453b7fe640 1 -- 172.21.15.73:0/1786244454 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45400c54d0 con 0x55ceb01a9080 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.773+0000 7f453b7fe640 1 -- 172.21.15.73:0/1786244454 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45400bd680 con 0x55ceb01a9080 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- 172.21.15.73:0/1786244454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 msgr2=0x55ceb01b13e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 crypto rx=0x7f45400bd150 tx=0x7f4540005b70 comp rx=0 tx=0).stop 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- 172.21.15.73:0/1786244454 shutdown_connections 2024-03-09T00:16:50.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b13e0 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 --2- 172.21.15.73:0/1786244454 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b0ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- 172.21.15.73:0/1786244454 >> 172.21.15.73:0/1786244454 conn(0x55ceb01a2e90 msgr2=0x55ceb01ae730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- 172.21.15.73:0/1786244454 shutdown_connections 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- 172.21.15.73:0/1786244454 wait complete. 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 -- start start 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.774+0000 7f455ac2a740 1 Processor -- start 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b92b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b7c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501ad4e0 con 0x55ceb01a9080 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501ae800 con 0x55ceb01b1c60 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45501adb40 con 0x55ceb01a8760 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 10 client.0 _ll_register_callbacks cb 0x7f4540003ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42552/0 (socket says 172.21.15.73:42552) 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 -- 172.21.15.73:0/3495476286 learned_addr learned my addr 172.21.15.73:0/3495476286 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f4555de9640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b92b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45565ea640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b7c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 -- 172.21.15.73:0/3495476286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 msgr2=0x55ceb01b92b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b92b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 -- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 msgr2=0x55ceb01b7c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b7c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.783 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 -- 172.21.15.73:0/3495476286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ceb01b8480 con 0x55ceb01a9080 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f4555de9640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45555e8640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f45400c5430 tx=0x7f45400c1120 comp rx=0 tx=0).ready entity=mon.0 client_cookie=38de3fb52ad90899 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45397fa640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f454004b070 con 0x55ceb01a9080 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f45400c54d0 con 0x55ceb01a9080 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.775+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f45400bc610 con 0x55ceb01a9080 2024-03-09T00:16:50.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f45400bcb20 con 0x55ceb01a9080 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 10 client.17196 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ceb0218c00 con 0x55ceb01a9080 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45400c54d0 con 0x55ceb01a9080 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f45400bc0c0 con 0x55ceb01a9080 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 10 client.17196 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 10 client.17196 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 20 client.17196 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:50.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb0218f00 0x55ceb021b320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 4 client.17196 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ceb0113690 con 0x55ceb0218f00 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.776+0000 7f4555de9640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb0218f00 0x55ceb021b320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f45397fa640 10 client.17196 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f4555de9640 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb0218f00 0x55ceb021b320 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=881dfc83b0e49fad server_cookie=fda55675ed0d366c in_seq=0 out_seq=0 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f45397fa640 1 -- 172.21.15.73:0/3495476286 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55ceb0113690 con 0x55ceb0218f00 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f45397fa640 10 client.17196 handle_command_reply: tid=0 2024-03-09T00:16:50.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 client.17196 shutdown 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 msgr2=0x55ceb01b7650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 crypto rx=0x7f45400c5430 tx=0x7f45400c1120 comp rx=0 tx=0).stop 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 shutdown_connections 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ceb0218f00 0x55ceb021b320 crc :-1 s=READY pgs=202 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ceb01b1c60 0x55ceb01b7c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ceb01a9080 0x55ceb01b7650 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f45397fa640 0 client.17196 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 --2- 172.21.15.73:0/3495476286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ceb01a8760 0x55ceb01b92b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.787 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 >> 172.21.15.73:0/3495476286 conn(0x55ceb01a2e90 msgr2=0x55ceb01acfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 shutdown_connections 2024-03-09T00:16:50.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 1 -- 172.21.15.73:0/3495476286 wait complete. 2024-03-09T00:16:50.788 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.777+0000 7f455ac2a740 20 client.17196 trim_cache size 0 max 16384 2024-03-09T00:16:50.788 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:50.789 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:16:50.789 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:16:50.789 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:50.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 msgr2=0x7f453c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.790 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 0x7f453c0a4e30 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f454801d000 tx=0x7f454801b000 comp rx=0 tx=0).stop 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 msgr2=0x7f455019ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 secure :-1 s=READY pgs=992 cs=0 l=1 rev1=1 crypto rx=0x7f4540000c00 tx=0x7f454002cc70 comp rx=0 tx=0).stop 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 shutdown_connections 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f4550197a20 0x7f4550197ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f453c0a2970 0x7f453c0a4e30 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f45501054a0 0x7f455019ddb0 unknown :-1 s=CLOSED pgs=992 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.791 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 --2- 172.21.15.73:0/2763461704 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f45501003d0 0x7f455019d870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 >> 172.21.15.73:0/2763461704 conn(0x7f45500772f0 msgr2=0x7f4550068850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 shutdown_connections 2024-03-09T00:16:50.792 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.778+0000 7f455804d640 1 -- 172.21.15.73:0/2763461704 wait complete. 2024-03-09T00:16:50.792 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:16:50.793 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:16:50.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 -- 172.21.15.73:0/2357348715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38100e10 msgr2=0x7fcf38104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/2357348715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38100e10 0x7fcf38104e50 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 crypto rx=0x7fcf20009900 tx=0x7fcf2002bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:50.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 -- 172.21.15.73:0/2357348715 shutdown_connections 2024-03-09T00:16:50.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/2357348715 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 0x7fcf3810d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/2357348715 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38100e10 0x7fcf38104e50 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/2357348715 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf38100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 -- 172.21.15.73:0/2357348715 >> 172.21.15.73:0/2357348715 conn(0x7fcf380772f0 msgr2=0x7fcf38077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 -- 172.21.15.73:0/2357348715 shutdown_connections 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.896+0000 7fcf3dc59640 1 -- 172.21.15.73:0/2357348715 wait complete. 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 Processor -- start 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 -- start start 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf3819d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 0x7fcf3819dda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf381105c0 con 0x7fcf38197a10 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38110440 con 0x7fcf38100460 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf3dc59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf38110740 con 0x7fcf38105530 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42566/0 (socket says 172.21.15.73:42566) 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 -- 172.21.15.73:0/540252598 learned_addr learned my addr 172.21.15.73:0/540252598 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf2ffff640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 0x7fcf3819dda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.903 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf377fe640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf3819d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 -- 172.21.15.73:0/540252598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 msgr2=0x7fcf3819dda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 0x7fcf3819dda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 -- 172.21.15.73:0/540252598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 msgr2=0x7fcf3819d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf3819d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 -- 172.21.15.73:0/540252598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf38198530 con 0x7fcf38197a10 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf377fe640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf3819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.897+0000 7fcf37fff640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000eaa0 tx=0x7fcf3000ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.904 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.898+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf3000cdb0 con 0x7fcf38197a10 2024-03-09T00:16:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.898+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcf30005020 con 0x7fcf38197a10 2024-03-09T00:16:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.898+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf38198820 con 0x7fcf38197a10 2024-03-09T00:16:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.898+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcf300116b0 con 0x7fcf38197a10 2024-03-09T00:16:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.898+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf381a4a70 con 0x7fcf38197a10 2024-03-09T00:16:50.905 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.899+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf380680e0 con 0x7fcf38197a10 2024-03-09T00:16:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.900+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fcf30005560 con 0x7fcf38197a10 2024-03-09T00:16:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.901+0000 7fcf357fa640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.901+0000 7fcf377fe640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.901+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fcf300cf8d0 con 0x7fcf38197a10 2024-03-09T00:16:50.906 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.901+0000 7fcf377fe640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 0x7fcf240a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf38198b50 tx=0x7fcf28005e70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:50.907 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:50.904+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fcf30016030 con 0x7fcf38197a10 2024-03-09T00:16:51.115 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.117+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf38198f40 con 0x7fcf38197a10 2024-03-09T00:16:51.116 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.117+0000 7fcf357fa640 1 -- 172.21.15.73:0/540252598 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7fcf38198f40 con 0x7fcf38197a10 2024-03-09T00:16:51.117 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:16:51.117 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:16:51.117 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 msgr2=0x7fcf240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 0x7fcf240a4e30 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7fcf38198b50 tx=0x7fcf28005e70 comp rx=0 tx=0).stop 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 msgr2=0x7fcf38197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 crypto rx=0x7fcf3000eaa0 tx=0x7fcf3000ef60 comp rx=0 tx=0).stop 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 shutdown_connections 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fcf240a2970 0x7fcf240a4e30 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.119 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fcf38197a10 0x7fcf38197ec0 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fcf38105530 0x7fcf3819dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 --2- 172.21.15.73:0/540252598 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fcf38100460 0x7fcf3819d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 >> 172.21.15.73:0/540252598 conn(0x7fcf380772f0 msgr2=0x7fcf380687c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 shutdown_connections 2024-03-09T00:16:51.120 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.120+0000 7fcf3dc59640 1 -- 172.21.15.73:0/540252598 wait complete. 2024-03-09T00:16:51.130 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:16:51.130 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:16:51.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 -- 172.21.15.73:0/2158868328 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 msgr2=0x7f373810d080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2158868328 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373810d080 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 crypto rx=0x7f3730009ca0 tx=0x7f373002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 -- 172.21.15.73:0/2158868328 shutdown_connections 2024-03-09T00:16:51.238 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2158868328 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373810d080 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2158868328 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3738100d90 0x7f3738104dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2158868328 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f37381003e0 0x7f37381007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 -- 172.21.15.73:0/2158868328 >> 172.21.15.73:0/2158868328 conn(0x7f37380773a0 msgr2=0x7f37380777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 -- 172.21.15.73:0/2158868328 shutdown_connections 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.237+0000 7f373cd3b640 1 -- 172.21.15.73:0/2158868328 wait complete. 2024-03-09T00:16:51.239 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 Processor -- start 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 -- start start 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37381003e0 0x7f373819c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 0x7f3738196d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381105c0 con 0x7f37381054b0 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3738110440 con 0x7f3738196880 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373cd3b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3738110740 con 0x7f37381003e0 2024-03-09T00:16:51.240 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42592/0 (socket says 172.21.15.73:42592) 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 -- 172.21.15.73:0/2622105109 learned_addr learned my addr 172.21.15.73:0/2622105109 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f373659c640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37381003e0 0x7f373819c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f3736d9d640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 0x7f3738196d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 -- 172.21.15.73:0/2622105109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37381003e0 msgr2=0x7f373819c6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.241 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37381003e0 0x7f373819c6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 -- 172.21.15.73:0/2622105109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 msgr2=0x7f3738196d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 0x7f3738196d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 -- 172.21.15.73:0/2622105109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37381973a0 con 0x7f37381054b0 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f3736d9d640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 0x7f3738196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.238+0000 7f372ffff640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f372000d920 tx=0x7f372000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.239+0000 7f372f7fe640 1 -- 172.21.15.73:0/2622105109 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372001b070 con 0x7f37381054b0 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.239+0000 7f372f7fe640 1 -- 172.21.15.73:0/2622105109 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f372000edc0 con 0x7f37381054b0 2024-03-09T00:16:51.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.239+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3738197690 con 0x7f37381054b0 2024-03-09T00:16:51.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.239+0000 7f372f7fe640 1 -- 172.21.15.73:0/2622105109 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37200147e0 con 0x7f37381054b0 2024-03-09T00:16:51.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.239+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37380687a0 con 0x7f37381054b0 2024-03-09T00:16:51.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.241+0000 7f372f7fe640 1 -- 172.21.15.73:0/2622105109 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f372001e070 con 0x7f37381054b0 2024-03-09T00:16:51.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.241+0000 7f373f9f8740 1 Processor -- start 2024-03-09T00:16:51.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.241+0000 7f373f9f8740 1 -- start start 2024-03-09T00:16:51.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.241+0000 7f372f7fe640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 0x7f37240a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b68ac0 0x556247b71160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b716a0 0x556247b73a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247b2f360 con 0x556247b681a0 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247acbec0 con 0x556247b716a0 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373f9f8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247a74d30 con 0x556247b68ac0 2024-03-09T00:16:51.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f3736d9d640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 0x7f37240a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f372f7fe640 1 -- 172.21.15.73:0/2622105109 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f37200d0890 con 0x7f37381054b0 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42594/0 (socket says 172.21.15.73:42594) 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 -- 172.21.15.73:0/903396430 learned_addr learned my addr 172.21.15.73:0/903396430 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f3736d9d640 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b716a0 0x556247b73a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 -- 172.21.15.73:0/903396430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b68ac0 msgr2=0x556247b71160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b68ac0 0x556247b71160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 -- 172.21.15.73:0/903396430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b716a0 msgr2=0x556247b73a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b716a0 0x556247b73a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 -- 172.21.15.73:0/903396430 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556247a73bc0 con 0x556247b681a0 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f3736d9d640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 0x7f37240a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3730032530 tx=0x7f37300099e0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f373659c640 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f3728009c20 tx=0x7f37280117b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8cc4067c7ffdf16c server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f372cff9640 1 -- 172.21.15.73:0/903396430 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728012450 con 0x556247b681a0 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f372cff9640 1 -- 172.21.15.73:0/903396430 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3728012ad0 con 0x556247b681a0 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.242+0000 7f372cff9640 1 -- 172.21.15.73:0/903396430 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f372800fb40 con 0x556247b681a0 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- 172.21.15.73:0/903396430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 msgr2=0x556247b68580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 crypto rx=0x7f3728009c20 tx=0x7f37280117b0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- 172.21.15.73:0/903396430 shutdown_connections 2024-03-09T00:16:51.254 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b716a0 0x556247b73a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b68ac0 0x556247b71160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 --2- 172.21.15.73:0/903396430 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b68580 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- 172.21.15.73:0/903396430 >> 172.21.15.73:0/903396430 conn(0x556247b628d0 msgr2=0x556247b62cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- 172.21.15.73:0/903396430 shutdown_connections 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- 172.21.15.73:0/903396430 wait complete. 2024-03-09T00:16:51.255 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.243+0000 7f373f9f8740 1 -- start start 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 Processor -- start 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 0x556247b6e630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6f0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247a6a4a0 con 0x556247b68ac0 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247ac7f60 con 0x556247b681a0 2024-03-09T00:16:51.256 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x556247b788a0 con 0x556247b716a0 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 10 client.0 _ll_register_callbacks cb 0x556247ac5d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42608/0 (socket says 172.21.15.73:42608) 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 -- 172.21.15.73:0/2204093144 learned_addr learned my addr 172.21.15.73:0/2204093144 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373659c640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 0x556247b6e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 -- 172.21.15.73:0/2204093144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 msgr2=0x556247b6f0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.257 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6f0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 -- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 msgr2=0x556247b6e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 0x556247b6e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 -- 172.21.15.73:0/2204093144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5562477e03f0 con 0x556247b68ac0 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373659c640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 0x556247b6e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f372ffff640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f37200ce1f0 tx=0x7f372009d310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9cee3f56c74a40aa server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f371dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37200a53b0 con 0x556247b68ac0 2024-03-09T00:16:51.258 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f3720087450 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.244+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3728012ad0 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3720099280 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 10 client.17214 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5562478aa7d0 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3728012ad0 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f372000e040 con 0x556247b68ac0 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 10 client.17214 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:51.259 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 10 client.17214 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 20 client.17214 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247b85c60 0x556247b88050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 4 client.17214 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556247ad9400 con 0x556247b85c60 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.245+0000 7f373659c640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247b85c60 0x556247b88050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f371dffb640 10 client.17214 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:51.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373659c640 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247b85c60 0x556247b88050 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d66485b1697be636 server_cookie=b931c26dfee348f3 in_seq=0 out_seq=0 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f371dffb640 1 -- 172.21.15.73:0/2204093144 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x556247ad9400 con 0x556247b85c60 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f371dffb640 10 client.17214 handle_command_reply: tid=0 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 client.17214 shutdown 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 msgr2=0x556247b6eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 crypto rx=0x7f37200ce1f0 tx=0x7f372009d310 comp rx=0 tx=0).stop 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 shutdown_connections 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247b85c60 0x556247b88050 crc :-1 s=READY pgs=203 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6f0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b68ac0 0x556247b6eb70 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f371dffb640 0 client.17214 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 --2- 172.21.15.73:0/2204093144 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b681a0 0x556247b6e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.246+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 >> 172.21.15.73:0/2204093144 conn(0x556247b628d0 msgr2=0x556247b64620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.247+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 shutdown_connections 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.247+0000 7f373f9f8740 1 -- 172.21.15.73:0/2204093144 wait complete. 2024-03-09T00:16:51.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.247+0000 7f373f9f8740 20 client.17214 trim_cache size 0 max 16384 2024-03-09T00:16:51.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 Processor -- start 2024-03-09T00:16:51.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 -- start start 2024-03-09T00:16:51.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b6d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6b890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381aa6c0 con 0x556247b681a0 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381acc60 con 0x556247b68ac0 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373f9f8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381aacc0 con 0x556247b716a0 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373659c640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42618/0 (socket says 172.21.15.73:42618) 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.258+0000 7f373659c640 1 -- 172.21.15.73:0/1998762927 learned_addr learned my addr 172.21.15.73:0/1998762927 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:51.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 -- 172.21.15.73:0/1998762927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 msgr2=0x556247b6b890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6b890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 -- 172.21.15.73:0/1998762927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 msgr2=0x556247b6d610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f372ffff640 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b6d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b6d610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 -- 172.21.15.73:0/1998762927 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556247b688f0 con 0x556247b681a0 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f372ffff640 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b6d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373659c640 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3728009c20 tx=0x7f372800f440 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7d3bbffac684b842 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f371ffff640 1 -- 172.21.15.73:0/1998762927 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3728005240 con 0x556247b681a0 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f371ffff640 1 -- 172.21.15.73:0/1998762927 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f372802a070 con 0x556247b681a0 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f371ffff640 1 -- 172.21.15.73:0/1998762927 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37280035f0 con 0x556247b681a0 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 -- 172.21.15.73:0/1998762927 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 msgr2=0x556247b70d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 secure :-1 s=READY pgs=1003 cs=0 l=1 rev1=1 crypto rx=0x7f3728009c20 tx=0x7f372800f440 comp rx=0 tx=0).stop 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 -- 172.21.15.73:0/1998762927 shutdown_connections 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b716a0 0x556247b6b890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b6d610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1998762927 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b681a0 0x556247b70d60 unknown :-1 s=CLOSED pgs=1003 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.271 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.259+0000 7f373f9f8740 1 -- 172.21.15.73:0/1998762927 >> 172.21.15.73:0/1998762927 conn(0x556247b628d0 msgr2=0x556247b6cf00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- 172.21.15.73:0/1998762927 shutdown_connections 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- 172.21.15.73:0/1998762927 wait complete. 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- start start 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 Processor -- start 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b681a0 0x556247b7b590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b7bad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381ab2e0 con 0x556247b716a0 2024-03-09T00:16:51.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381ac600 con 0x556247b68ac0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37381ab940 con 0x556247b681a0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373f9f8740 10 client.0 _ll_register_callbacks cb 0x7f3728009670 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42622/0 (socket says 172.21.15.73:42622) 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 -- 172.21.15.73:0/1865837237 learned_addr learned my addr 172.21.15.73:0/1865837237 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f372ffff640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b7bad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f373659c640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b681a0 0x556247b7b590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 -- 172.21.15.73:0/1865837237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b681a0 msgr2=0x556247b7b590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b681a0 0x556247b7b590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 -- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 msgr2=0x556247b7bad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b7bad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f3736d9d640 1 -- 172.21.15.73:0/1865837237 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556247bdc220 con 0x556247b716a0 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.260+0000 7f372ffff640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b7bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f3736d9d640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3730009ca0 tx=0x7f3730004aa0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79d30afb35a1a468 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:51.274 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3730039070 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f3730003b80 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f372802a070 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3730059050 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 10 client.17220 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x556247bdd490 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f372802a070 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f3730054030 con 0x556247b716a0 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 10 client.17220 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:51.275 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 10 client.17220 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 20 client.17220 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247bddc10 0x556247be0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 4 client.17220 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.261+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556247ad3750 con 0x556247bddc10 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373659c640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247bddc10 0x556247be0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373659c640 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247bddc10 0x556247be0030 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=797b14661905dc56 server_cookie=b53f7babe0a58ad3 in_seq=0 out_seq=0 2024-03-09T00:16:51.276 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f371dffb640 10 client.17220 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f371dffb640 1 -- 172.21.15.73:0/1865837237 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556247ad3750 con 0x556247bddc10 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f371dffb640 10 client.17220 handle_command_reply: tid=0 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 client.17220 shutdown 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 msgr2=0x556247b7c010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 crypto rx=0x7f3730009ca0 tx=0x7f3730004aa0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 shutdown_connections 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x556247bddc10 0x556247be0030 crc :-1 s=READY pgs=204 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x556247b716a0 0x556247b7c010 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.277 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x556247b68ac0 0x556247b7bad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f371dffb640 0 client.17220 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:51.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 --2- 172.21.15.73:0/1865837237 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x556247b681a0 0x556247b7b590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 >> 172.21.15.73:0/1865837237 conn(0x556247b628d0 msgr2=0x556247b61100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 shutdown_connections 2024-03-09T00:16:51.278 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.262+0000 7f373f9f8740 1 -- 172.21.15.73:0/1865837237 wait complete. 2024-03-09T00:16:51.279 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373f9f8740 20 client.17220 trim_cache size 0 max 16384 2024-03-09T00:16:51.279 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 msgr2=0x7f37240a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 0x7f37240a4e30 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f3730032530 tx=0x7f37300099e0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 msgr2=0x7f373819cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 crypto rx=0x7f372000d920 tx=0x7f372000dde0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 shutdown_connections 2024-03-09T00:16:51.281 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3738196880 0x7f3738196d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f37240a2970 0x7f37240a4e30 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f37381054b0 0x7f373819cc10 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 --2- 172.21.15.73:0/2622105109 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f37381003e0 0x7f373819c6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 >> 172.21.15.73:0/2622105109 conn(0x7f37380773a0 msgr2=0x7f37380fe0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 shutdown_connections 2024-03-09T00:16:51.282 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:51.263+0000 7f373cd3b640 1 -- 172.21.15.73:0/2622105109 wait complete. 2024-03-09T00:16:51.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:51 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/540252598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:51.514 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:51 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/540252598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:51.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:51 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/540252598' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:16:52.284 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-09T00:16:52.284 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:16:52.284 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- 172.21.15.73:0/3175417488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d40ffa60 msgr2=0x7ff5d40ffe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 --2- 172.21.15.73:0/3175417488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d40ffa60 0x7ff5d40ffe40 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 crypto rx=0x7ff5d0009900 tx=0x7ff5d002bc50 comp rx=0 tx=0).stop 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- 172.21.15.73:0/3175417488 shutdown_connections 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 --2- 172.21.15.73:0/3175417488 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d4106670 0x7ff5d410e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 --2- 172.21.15.73:0/3175417488 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4100410 0x7ff5d4105f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.405 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 --2- 172.21.15.73:0/3175417488 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d40ffa60 0x7ff5d40ffe40 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- 172.21.15.73:0/3175417488 >> 172.21.15.73:0/3175417488 conn(0x7ff5d40767b0 msgr2=0x7ff5d40fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- 172.21.15.73:0/3175417488 shutdown_connections 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- 172.21.15.73:0/3175417488 wait complete. 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 Processor -- start 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.404+0000 7ff5db620640 1 -- start start 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d40ffa60 0x7ff5d419d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 0x7ff5d4197f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4111710 con 0x7ff5d4106670 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4111590 con 0x7ff5d4197a70 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5db620640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d4111890 con 0x7ff5d40ffa60 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42646/0 (socket says 172.21.15.73:42646) 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 -- 172.21.15.73:0/3843787904 learned_addr learned my addr 172.21.15.73:0/3843787904 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d40ffa60 0x7ff5d419d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 0x7ff5d4197f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 -- 172.21.15.73:0/3843787904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d40ffa60 msgr2=0x7ff5d419d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d40ffa60 0x7ff5d419d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 -- 172.21.15.73:0/3843787904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 msgr2=0x7ff5d4197f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 0x7ff5d4197f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 -- 172.21.15.73:0/3843787904 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff5d4198590 con 0x7ff5d4106670 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 0x7ff5d4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:16:52.408 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4006fd0 tx=0x7ff5c400b8e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5ca7fc640 1 -- 172.21.15.73:0/3843787904 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c4011830 con 0x7ff5d4106670 2024-03-09T00:16:52.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.405+0000 7ff5ca7fc640 1 -- 172.21.15.73:0/3843787904 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5c4016070 con 0x7ff5d4106670 2024-03-09T00:16:52.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.406+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5d4198880 con 0x7ff5d4106670 2024-03-09T00:16:52.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.406+0000 7ff5ca7fc640 1 -- 172.21.15.73:0/3843787904 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5c400f2e0 con 0x7ff5d4106670 2024-03-09T00:16:52.409 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.406+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5d41a4ad0 con 0x7ff5d4106670 2024-03-09T00:16:52.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.408+0000 7ff5ca7fc640 1 -- 172.21.15.73:0/3843787904 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff5c401b460 con 0x7ff5d4106670 2024-03-09T00:16:52.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.408+0000 7ff5de204740 1 Processor -- start 2024-03-09T00:16:52.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.408+0000 7ff5de204740 1 -- start start 2024-03-09T00:16:52.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.416 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d18d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d1b670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7c73b40 con 0x5620f7d0fd80 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7c1cd80 con 0x5620f7d106a0 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5de204740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7c6fbe0 con 0x5620f7d19280 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5ca7fc640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 0x7ff5c00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42648/0 (socket says 172.21.15.73:42648) 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/4096987833 learned_addr learned my addr 172.21.15.73:0/4096987833 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.417 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 0x7ff5c00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5ca7fc640 1 -- 172.21.15.73:0/3843787904 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7ff5c40ce820 con 0x7ff5d4106670 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d18d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d1b670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/4096987833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 msgr2=0x5620f7d1b670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d1b670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/4096987833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 msgr2=0x5620f7d18d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d18d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/4096987833 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620f7a507e0 con 0x5620f7d0fd80 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d18d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc012e20 tx=0x7ff5cc00f730 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c6f2e2944c2a9757 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.409+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 0x7ff5c00a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00afd0 tx=0x7ff5cc01a000 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5bf7fe640 1 -- 172.21.15.73:0/4096987833 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc020070 con 0x5620f7d0fd80 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5bf7fe640 1 -- 172.21.15.73:0/4096987833 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc00ac00 con 0x5620f7d0fd80 2024-03-09T00:16:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5bf7fe640 1 -- 172.21.15.73:0/4096987833 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc01cdb0 con 0x5620f7d0fd80 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 -- 172.21.15.73:0/4096987833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 msgr2=0x5620f7d10160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 secure :-1 s=READY pgs=1007 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc012e20 tx=0x7ff5cc00f730 comp rx=0 tx=0).stop 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 -- 172.21.15.73:0/4096987833 shutdown_connections 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d1b670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d18d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 --2- 172.21.15.73:0/4096987833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d10160 unknown :-1 s=CLOSED pgs=1007 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 -- 172.21.15.73:0/4096987833 >> 172.21.15.73:0/4096987833 conn(0x5620f7d0a4b0 msgr2=0x5620f7d0a890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.420 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 -- 172.21.15.73:0/4096987833 shutdown_connections 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.410+0000 7ff5de204740 1 -- 172.21.15.73:0/4096987833 wait complete. 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 -- start start 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 Processor -- start 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d16070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d16790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7d1c010 con 0x5620f7d0fd80 2024-03-09T00:16:52.421 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7c301c0 con 0x5620f7d106a0 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x5620f7d1c190 con 0x5620f7d19280 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5de204740 10 client.0 _ll_register_callbacks cb 0x5620f7c1f7a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5d93bc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5d93bc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42658/0 (socket says 172.21.15.73:42658) 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.411+0000 7ff5d93bc640 1 -- 172.21.15.73:0/4240000104 learned_addr learned my addr 172.21.15.73:0/4240000104 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d16070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d16790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.422 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 -- 172.21.15.73:0/4240000104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 msgr2=0x5620f7d16790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d16790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 -- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 msgr2=0x5620f7d16070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d16070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 -- 172.21.15.73:0/4240000104 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620f7ab0f70 con 0x5620f7d0fd80 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d16070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff5d002c190 tx=0x7ff5d0005eb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4e6b703a257d75cd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5d00046c0 con 0x5620f7d0fd80 2024-03-09T00:16:52.423 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7ff5d0004860 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5cc00ac00 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5d0004a60 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5de204740 10 client.17232 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5620f7948170 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.412+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc00ac00 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff5d00037f0 con 0x5620f7d0fd80 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 10 client.17232 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:52.424 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 10 client.17232 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 20 client.17232 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d29370 0x5620f7d2b760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 4 client.17232 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5620f7c49960 con 0x5620f7d29370 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d29370 0x5620f7d2b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5bcff9640 10 client.17232 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.413+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d29370 0x5620f7d2b760 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca73ca1345e382a5 server_cookie=b8e7bf3709d935c1 in_seq=0 out_seq=0 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5bcff9640 1 -- 172.21.15.73:0/4240000104 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x7ff5c4065a90 con 0x5620f7d29370 2024-03-09T00:16:52.425 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5bcff9640 10 client.17232 handle_command_reply: tid=0 2024-03-09T00:16:52.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 client.17232 shutdown 2024-03-09T00:16:52.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 msgr2=0x5620f7d18ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.426 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 secure :-1 s=READY pgs=1008 cs=0 l=1 rev1=1 crypto rx=0x7ff5d002c190 tx=0x7ff5d0005eb0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 shutdown_connections 2024-03-09T00:16:52.427 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d29370 0x5620f7d2b760 crc :-1 s=READY pgs=205 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d19280 0x5620f7d16790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d106a0 0x5620f7d16070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 --2- 172.21.15.73:0/4240000104 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d0fd80 0x5620f7d18ce0 unknown :-1 s=CLOSED pgs=1008 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5bcff9640 0 client.17232 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 >> 172.21.15.73:0/4240000104 conn(0x5620f7d0a4b0 msgr2=0x5620f7d0c200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.428 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 shutdown_connections 2024-03-09T00:16:52.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 1 -- 172.21.15.73:0/4240000104 wait complete. 2024-03-09T00:16:52.429 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.414+0000 7ff5de204740 20 client.17232 trim_cache size 0 max 16384 2024-03-09T00:16:52.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 Processor -- start 2024-03-09T00:16:52.437 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 -- start start 2024-03-09T00:16:52.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d15090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d187a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.438 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41ac960 con 0x5620f7d19280 2024-03-09T00:16:52.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41aef00 con 0x5620f7d0fd80 2024-03-09T00:16:52.439 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5de204740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41acf60 con 0x5620f7d106a0 2024-03-09T00:16:52.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.429+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42670/0 (socket says 172.21.15.73:42670) 2024-03-09T00:16:52.440 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/860404655 learned_addr learned my addr 172.21.15.73:0/860404655 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d15090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/860404655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 msgr2=0x5620f7d187a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.441 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d8bbb640 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d187a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d187a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/860404655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 msgr2=0x5620f7d15090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d15090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/860404655 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5620f7d14600 con 0x5620f7d19280 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00a670 tx=0x7ff5cc00a6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=79dd1a855e7dc056 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5beffd640 1 -- 172.21.15.73:0/860404655 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc032070 con 0x5620f7d19280 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5beffd640 1 -- 172.21.15.73:0/860404655 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc019b70 con 0x5620f7d19280 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.430+0000 7ff5beffd640 1 -- 172.21.15.73:0/860404655 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc020040 con 0x5620f7d19280 2024-03-09T00:16:52.442 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- 172.21.15.73:0/860404655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 msgr2=0x5620f7d18ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 secure :-1 s=READY pgs=1009 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00a670 tx=0x7ff5cc00a6a0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- 172.21.15.73:0/860404655 shutdown_connections 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d18ce0 unknown :-1 s=CLOSED pgs=1009 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d187a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 --2- 172.21.15.73:0/860404655 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d15090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- 172.21.15.73:0/860404655 >> 172.21.15.73:0/860404655 conn(0x5620f7d0a4b0 msgr2=0x5620f7d14980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- 172.21.15.73:0/860404655 shutdown_connections 2024-03-09T00:16:52.443 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- 172.21.15.73:0/860404655 wait complete. 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 -- start start 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.431+0000 7ff5de204740 1 Processor -- start 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d78d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d79290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41ad580 con 0x5620f7d19280 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41ae8a0 con 0x5620f7d0fd80 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5d41adbe0 con 0x5620f7d106a0 2024-03-09T00:16:52.444 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 10 client.0 _ll_register_callbacks cb 0x7ff5cc00ae60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42684/0 (socket says 172.21.15.73:42684) 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/580130152 learned_addr learned my addr 172.21.15.73:0/580130152 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d78d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/580130152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 msgr2=0x5620f7d79290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d79290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 msgr2=0x5620f7d78d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d78d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.445 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 -- 172.21.15.73:0/580130152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5620f7d20170 con 0x5620f7d19280 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5d9bbd640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc030040 tx=0x7ff5cc012490 comp rx=0 tx=0).ready entity=mon.0 client_cookie=97336eb186853a72 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5bcff9640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff5cc03d060 con 0x5620f7d19280 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff5cc019b70 con 0x5620f7d19280 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7ff5cc038680 con 0x5620f7d19280 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff5cc032070 con 0x5620f7d19280 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 10 client.17238 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:52.446 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.432+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5620f7d81660 con 0x5620f7d19280 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff5cc019b70 con 0x5620f7d19280 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7ff5cc01b070 con 0x5620f7d19280 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 10 client.17238 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 10 client.17238 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 20 client.17238 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d81b30 0x5620f7d83f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 4 client.17238 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5620f7c35080 con 0x5620f7d81b30 2024-03-09T00:16:52.447 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d81b30 0x5620f7d83f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5bcff9640 10 client.17238 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.433+0000 7ff5d93bc640 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d81b30 0x5620f7d83f20 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9873f941ea153b00 server_cookie=b8303856eac79845 in_seq=0 out_seq=0 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5bcff9640 1 -- 172.21.15.73:0/580130152 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5620f7c35080 con 0x5620f7d81b30 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5bcff9640 10 client.17238 handle_command_reply: tid=0 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 client.17238 shutdown 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 msgr2=0x5620f7d1f940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 secure :-1 s=READY pgs=1010 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc030040 tx=0x7ff5cc012490 comp rx=0 tx=0).stop 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 shutdown_connections 2024-03-09T00:16:52.448 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x5620f7d81b30 0x5620f7d83f20 crc :-1 s=READY pgs=206 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x5620f7d19280 0x5620f7d1f940 unknown :-1 s=CLOSED pgs=1010 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x5620f7d106a0 0x5620f7d79290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 --2- 172.21.15.73:0/580130152 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x5620f7d0fd80 0x5620f7d78d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 >> 172.21.15.73:0/580130152 conn(0x5620f7d0a4b0 msgr2=0x5620f7d08bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5bcff9640 0 client.17238 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 shutdown_connections 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 1 -- 172.21.15.73:0/580130152 wait complete. 2024-03-09T00:16:52.449 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.435+0000 7ff5de204740 20 client.17238 trim_cache size 0 max 16384 2024-03-09T00:16:52.450 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:52.450 INFO:teuthology.orchestra.run.smithi073.stdout: "return_code": 0, 2024-03-09T00:16:52.450 INFO:teuthology.orchestra.run.smithi073.stdout: "scrub_tag": "49190483-2241-461c-a611-c3c3bbea5d72", 2024-03-09T00:16:52.450 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": "asynchronous" 2024-03-09T00:16:52.450 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:52.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 msgr2=0x7ff5c00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.451 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 0x7ff5c00a4e30 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7ff5cc00afd0 tx=0x7ff5cc01a000 comp rx=0 tx=0).stop 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 msgr2=0x7ff5d419de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 crypto rx=0x7ff5c4006fd0 tx=0x7ff5c400b8e0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 shutdown_connections 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff5d4197a70 0x7ff5d4197f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff5c00a2970 0x7ff5c00a4e30 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.452 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff5d4106670 0x7ff5d419de00 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 --2- 172.21.15.73:0/3843787904 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff5d40ffa60 0x7ff5d419d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 >> 172.21.15.73:0/3843787904 conn(0x7ff5d40767b0 msgr2=0x7ff5d4104720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 shutdown_connections 2024-03-09T00:16:52.453 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.436+0000 7ff5db620640 1 -- 172.21.15.73:0/3843787904 wait complete. 2024-03-09T00:16:52.454 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:52 smithi073 ceph-mon[32309]: pgmap v311: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:52.455 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:16:52.455 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:16:52.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:52 smithi102 ceph-mon[36106]: pgmap v311: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:52.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 -- 172.21.15.73:0/159968691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 msgr2=0x7f7a6410e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/159968691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6410e170 secure :-1 s=READY pgs=1011 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009ca0 tx=0x7f7a5c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 -- 172.21.15.73:0/159968691 shutdown_connections 2024-03-09T00:16:52.566 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/159968691 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6410e170 unknown :-1 s=CLOSED pgs=1011 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/159968691 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a640690f0 0x7f7a64105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.566+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/159968691 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a64068740 0x7f7a64068b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.567+0000 7f7a68fe6640 1 -- 172.21.15.73:0/159968691 >> 172.21.15.73:0/159968691 conn(0x7f7a640772f0 msgr2=0x7f7a64077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.567+0000 7f7a68fe6640 1 -- 172.21.15.73:0/159968691 shutdown_connections 2024-03-09T00:16:52.567 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.567+0000 7f7a68fe6640 1 -- 172.21.15.73:0/159968691 wait complete. 2024-03-09T00:16:52.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.567+0000 7f7a68fe6640 1 Processor -- start 2024-03-09T00:16:52.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.567+0000 7f7a68fe6640 1 -- start start 2024-03-09T00:16:52.569 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 0x7f7a6419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641979e0 0x7f7a64197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a64111660 con 0x7f7a641065a0 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641114e0 con 0x7f7a641979e0 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a68fe6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641117e0 con 0x7f7a64068740 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 0x7f7a6419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.570 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 0x7f7a6419d7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47688/0 (socket says 172.21.15.73:47688) 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a62d9d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641979e0 0x7f7a64197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a6259c640 1 -- 172.21.15.73:0/1558425021 learned_addr learned my addr 172.21.15.73:0/1558425021 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1558425021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 msgr2=0x7f7a6419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 0x7f7a6419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1558425021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641979e0 msgr2=0x7f7a64197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641979e0 0x7f7a64197e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1558425021 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a64198500 con 0x7f7a641065a0 2024-03-09T00:16:52.571 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.568+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6419dd20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00e9c0 tx=0x7f7a4c00ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.569+0000 7f7a5b7fe640 1 -- 172.21.15.73:0/1558425021 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c00ccc0 con 0x7f7a641065a0 2024-03-09T00:16:52.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.569+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a641987f0 con 0x7f7a641065a0 2024-03-09T00:16:52.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.569+0000 7f7a5b7fe640 1 -- 172.21.15.73:0/1558425021 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c005020 con 0x7f7a641065a0 2024-03-09T00:16:52.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.569+0000 7f7a5b7fe640 1 -- 172.21.15.73:0/1558425021 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c004a00 con 0x7f7a641065a0 2024-03-09T00:16:52.572 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.569+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a641a49f0 con 0x7f7a641065a0 2024-03-09T00:16:52.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.571+0000 7f7a5b7fe640 1 -- 172.21.15.73:0/1558425021 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7a4c0051d0 con 0x7f7a641065a0 2024-03-09T00:16:52.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 Processor -- start 2024-03-09T00:16:52.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 -- start start 2024-03-09T00:16:52.582 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a5b7fe640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 0x7f7a500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53075b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee5307e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53081050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fd8e00 con 0x55ee53076080 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.572+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 0x7f7a500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52f81b10 con 0x55ee53075760 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52fd4ea0 con 0x55ee5307ec60 2024-03-09T00:16:52.583 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5b7fe640 1 -- 172.21.15.73:0/1558425021 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f7a4c0cf100 con 0x7f7a641065a0 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53075b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53075b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:44204/0 (socket says 172.21.15.73:44204) 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee5307e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a6259c640 1 -- 172.21.15.73:0/818936833 learned_addr learned my addr 172.21.15.73:0/818936833 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 -- 172.21.15.73:0/818936833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 msgr2=0x55ee53081050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53081050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 -- 172.21.15.73:0/818936833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 msgr2=0x55ee53075b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53075b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.584 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 -- 172.21.15.73:0/818936833 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee52db77d0 con 0x55ee53076080 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 0x7f7a500a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c005e30 tx=0x7f7a5c005f10 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee5307e720 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0cc460 tx=0x7f7a4c09bb30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4a16d54b1a8cff server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.573+0000 7f7a58ff9640 1 -- 172.21.15.73:0/818936833 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c01c040 con 0x55ee53076080 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a58ff9640 1 -- 172.21.15.73:0/818936833 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0a3b10 con 0x55ee53076080 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a58ff9640 1 -- 172.21.15.73:0/818936833 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c0a3680 con 0x55ee53076080 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/818936833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 msgr2=0x55ee5307e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee5307e720 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c0cc460 tx=0x7f7a4c09bb30 comp rx=0 tx=0).stop 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/818936833 shutdown_connections 2024-03-09T00:16:52.585 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53081050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee5307e720 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/818936833 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53075b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.574+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/818936833 >> 172.21.15.73:0/818936833 conn(0x55ee5306fe90 msgr2=0x55ee53070270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/818936833 shutdown_connections 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/818936833 wait complete. 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- start start 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 Processor -- start 2024-03-09T00:16:52.586 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee5307bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee53081a50 con 0x55ee5307ec60 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee52f950a0 con 0x55ee53076080 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ee53081bd0 con 0x55ee53075760 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6bbc6740 10 client.0 _ll_register_callbacks cb 0x55ee52f94a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.575+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.587 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.179:3300/0 says I am v2:172.21.15.73:47710/0 (socket says 172.21.15.73:47710) 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.102:3300/0 says I am v2:172.21.15.73:44216/0 (socket says 172.21.15.73:44216) 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 -- 172.21.15.73:0/2099961142 learned_addr learned my addr 172.21.15.73:0/2099961142 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee5307bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 -- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 msgr2=0x55ee5307b740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 -- 172.21.15.73:0/2099961142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 msgr2=0x55ee5307bc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee5307bc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 -- 172.21.15.73:0/2099961142 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee52ced3f0 con 0x55ee53075760 2024-03-09T00:16:52.588 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee5307bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6259c640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a540127f0 tx=0x7f7a54012cb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f832f7a35378107d server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.179:3300/0 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mon.1 v2:172.21.15.179:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a54004020 con 0x55ee53075760 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a4c0a3b10 con 0x55ee53075760 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.576+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mon.1 v2:172.21.15.179:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f7a540041c0 con 0x55ee53075760 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mon.1 v2:172.21.15.179:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a54025050 con 0x55ee53075760 2024-03-09T00:16:52.589 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 10 client.26368 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ee52cad0f0 con 0x55ee53075760 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mon.1 v2:172.21.15.179:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a4c0a3b10 con 0x55ee53075760 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mon.1 v2:172.21.15.179:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a54003980 con 0x55ee53075760 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 10 client.26368 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 10 client.26368 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 20 client.26368 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee5308e7a0 0x55ee53090bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.590 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 4 client.26368 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ee52fe6340 con 0x55ee5308e7a0 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.577+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee5308e7a0 0x55ee53090bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.578+0000 7f7a49ffb640 10 client.26368 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.578+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee5308e7a0 0x55ee53090bb0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ca7e30241b1fb69a server_cookie=1851d3abe6f88e51 in_seq=0 out_seq=0 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a49ffb640 1 -- 172.21.15.73:0/2099961142 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x55ee52fe6340 con 0x55ee5308e7a0 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a49ffb640 10 client.26368 handle_command_reply: tid=0 2024-03-09T00:16:52.591 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 client.26368 shutdown 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 msgr2=0x55ee5307e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f7a540127f0 tx=0x7f7a54012cb0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 shutdown_connections 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee5308e7a0 0x55ee53090bb0 crc :-1 s=READY pgs=207 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee5307bc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.592 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307b740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/2099961142 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307e3e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a49ffb640 0 client.26368 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.593 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 >> 172.21.15.73:0/2099961142 conn(0x55ee5306fe90 msgr2=0x55ee53071be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 shutdown_connections 2024-03-09T00:16:52.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/2099961142 wait complete. 2024-03-09T00:16:52.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.579+0000 7f7a6bbc6740 20 client.26368 trim_cache size 0 max 16384 2024-03-09T00:16:52.596 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:52 smithi179 ceph-mon[36637]: pgmap v311: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 Processor -- start 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 -- start start 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307df80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641aa670 con 0x55ee5307ec60 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641acc10 con 0x55ee53076080 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641aac70 con 0x55ee53075760 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a62d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a6259c640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a62d9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42740/0 (socket says 172.21.15.73:42740) 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a62d9d640 1 -- 172.21.15.73:0/3312181286 learned_addr learned my addr 172.21.15.73:0/3312181286 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.596+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 -- 172.21.15.73:0/3312181286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 msgr2=0x55ee5307b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 -- 172.21.15.73:0/3312181286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 msgr2=0x55ee5307df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 -- 172.21.15.73:0/3312181286 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ee53075eb0 con 0x55ee5307ec60 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009ca0 tx=0x7f7a5c040000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c8344ef9a38bde78 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a4bfff640 1 -- 172.21.15.73:0/3312181286 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c051070 con 0x55ee5307ec60 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a4bfff640 1 -- 172.21.15.73:0/3312181286 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c004040 con 0x55ee5307ec60 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.597+0000 7f7a4bfff640 1 -- 172.21.15.73:0/3312181286 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a5c039040 con 0x55ee5307ec60 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/3312181286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 msgr2=0x55ee53085e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 secure :-1 s=READY pgs=1014 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c009ca0 tx=0x7f7a5c040000 comp rx=0 tx=0).stop 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/3312181286 shutdown_connections 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee5307ec60 0x55ee53085e00 unknown :-1 s=CLOSED pgs=1014 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53076080 0x55ee5307df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/3312181286 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee53075760 0x55ee5307b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/3312181286 >> 172.21.15.73:0/3312181286 conn(0x55ee5306fe90 msgr2=0x55ee5308a8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/3312181286 shutdown_connections 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/3312181286 wait complete. 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 -- start start 2024-03-09T00:16:52.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.598+0000 7f7a6bbc6740 1 Processor -- start 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53085e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53077a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641ab290 con 0x55ee53076080 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641ac5b0 con 0x55ee53075760 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a641ab8f0 con 0x55ee5307ec60 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 10 client.0 _ll_register_callbacks cb 0x7f7a540098c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:16:52.606 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42742/0 (socket says 172.21.15.73:42742) 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1619134404 learned_addr learned my addr 172.21.15.73:0/1619134404 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6259c640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53085e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a62d9d640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53077a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1619134404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 msgr2=0x55ee53077a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53077a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 msgr2=0x55ee53085e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53085e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.607 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 -- 172.21.15.73:0/1619134404 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ee53078130 con 0x55ee53076080 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a5bfff640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c097ba0 tx=0x7f7a4c0096e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=31b2fddd75a670ce server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a49ffb640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a4c01c040 con 0x55ee53076080 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7a5c004040 con 0x55ee53076080 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f7a4c09f1b0 con 0x55ee53076080 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.599+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7a4c0a3b10 con 0x55ee53076080 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 10 client.17256 fetch_fsmap learned FSMap version 20 2024-03-09T00:16:52.608 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55ee530e5620 con 0x55ee53076080 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a5c004040 con 0x55ee53076080 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7f7a4c0ccbb0 con 0x55ee53076080 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 10 client.17256 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 10 client.17256 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 20 client.17256 populate_metadata read hostname 'smithi073' 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee530e5950 0x55ee530e7d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 4 client.17256 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55ee52fe0690 con 0x55ee530e5950 2024-03-09T00:16:52.609 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6259c640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee530e5950 0x55ee530e7d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a6259c640 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee530e5950 0x55ee530e7d40 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14d3a307d19a8f54 server_cookie=ea4417c68fdd3550 in_seq=0 out_seq=0 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.600+0000 7f7a49ffb640 10 client.17256 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a49ffb640 1 -- 172.21.15.73:0/1619134404 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x55ee52fe0690 con 0x55ee530e5950 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a49ffb640 10 client.17256 handle_command_reply: tid=0 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 client.17256 shutdown 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 msgr2=0x55ee53086350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c097ba0 tx=0x7f7a4c0096e0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.610 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 shutdown_connections 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55ee530e5950 0x55ee530e7d40 crc :-1 s=READY pgs=208 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x55ee5307ec60 0x55ee53077a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a49ffb640 0 client.17256 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x55ee53076080 0x55ee53086350 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 --2- 172.21.15.73:0/1619134404 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x55ee53075760 0x55ee53085e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 >> 172.21.15.73:0/1619134404 conn(0x55ee5306fe90 msgr2=0x55ee530765a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 shutdown_connections 2024-03-09T00:16:52.611 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 1 -- 172.21.15.73:0/1619134404 wait complete. 2024-03-09T00:16:52.612 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.601+0000 7f7a6bbc6740 20 client.17256 trim_cache size 0 max 16384 2024-03-09T00:16:52.612 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:16:52.612 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "scrub active (2393 inodes in the stack)", 2024-03-09T00:16:52.612 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": { 2024-03-09T00:16:52.612 INFO:teuthology.orchestra.run.smithi073.stdout: "49190483-2241-461c-a611-c3c3bbea5d72": { 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout: "path": "/", 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout: "tag": "49190483-2241-461c-a611-c3c3bbea5d72", 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout: "options": "recursive,force" 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:16:52.613 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 msgr2=0x7f7a500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 0x7f7a500a4e30 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f7a5c005e30 tx=0x7f7a5c005f10 comp rx=0 tx=0).stop 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 msgr2=0x7f7a6419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6419dd20 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 crypto rx=0x7f7a4c00e9c0 tx=0x7f7a4c00ee80 comp rx=0 tx=0).stop 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 shutdown_connections 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7a641979e0 0x7f7a64197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.615 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f7a500a2970 0x7f7a500a4e30 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7a641065a0 0x7f7a6419dd20 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 --2- 172.21.15.73:0/1558425021 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7a64068740 0x7f7a6419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:52.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 >> 172.21.15.73:0/1558425021 conn(0x7f7a640772f0 msgr2=0x7f7a640fe380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:52.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 shutdown_connections 2024-03-09T00:16:52.616 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:52.602+0000 7f7a68fe6640 1 -- 172.21.15.73:0/1558425021 wait complete. 2024-03-09T00:16:52.616 INFO:tasks.cephfs.filesystem:scrub status for tag:49190483-2241-461c-a611-c3c3bbea5d72 - {'path': '/', 'tag': '49190483-2241-461c-a611-c3c3bbea5d72', 'options': 'recursive,force'} 2024-03-09T00:16:53.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:53 smithi102 ceph-mon[36106]: scrub queued for path: / 2024-03-09T00:16:53.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:53 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:53 smithi102 ceph-mon[36106]: scrub summary: active paths [/] 2024-03-09T00:16:53.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:53 smithi102 ceph-mon[36106]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:53 smithi179 ceph-mon[36637]: scrub queued for path: / 2024-03-09T00:16:53.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:53 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:53 smithi179 ceph-mon[36637]: scrub summary: active paths [/] 2024-03-09T00:16:53.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:53 smithi179 ceph-mon[36637]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:53 smithi073 ceph-mon[32309]: scrub queued for path: / 2024-03-09T00:16:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:53 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:53 smithi073 ceph-mon[32309]: scrub summary: active paths [/] 2024-03-09T00:16:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:53 smithi073 ceph-mon[32309]: scrub summary: idle+waiting paths [/] 2024-03-09T00:16:53.816 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule status --fs=cephfs --path=/' 2024-03-09T00:16:53.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- 172.21.15.73:0/2171320701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100440 msgr2=0x7f10441008c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:53.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 --2- 172.21.15.73:0/2171320701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100440 0x7f10441008c0 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 crypto rx=0x7f102c009900 tx=0x7f102c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:53.929 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- 172.21.15.73:0/2171320701 shutdown_connections 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 --2- 172.21.15.73:0/2171320701 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100e00 0x7f104410d020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 --2- 172.21.15.73:0/2171320701 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100440 0x7f10441008c0 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 --2- 172.21.15.73:0/2171320701 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 0x7f1044105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- 172.21.15.73:0/2171320701 >> 172.21.15.73:0/2171320701 conn(0x7f1044077040 msgr2=0x7f1044077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- 172.21.15.73:0/2171320701 shutdown_connections 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- 172.21.15.73:0/2171320701 wait complete. 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 Processor -- start 2024-03-09T00:16:53.930 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.930+0000 7f104a560640 1 -- start start 2024-03-09T00:16:53.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 0x7f104419d7e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:53.933 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 0x7f1044197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1044110630 con 0x7f1044100e00 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10441104b0 con 0x7f1044100440 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f104a560640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10441107b0 con 0x7f1044105190 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42756/0 (socket says 172.21.15.73:42756) 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 -- 172.21.15.73:0/765878069 learned_addr learned my addr 172.21.15.73:0/765878069 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:53.934 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f1048afd640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 0x7f1044197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f1043fff640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 0x7f104419d7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 -- 172.21.15.73:0/765878069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 msgr2=0x7f1044197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 0x7f1044197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 -- 172.21.15.73:0/765878069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 msgr2=0x7f104419d7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 0x7f104419d7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 -- 172.21.15.73:0/765878069 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1044198110 con 0x7f1044100e00 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f1043fff640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 0x7f104419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.931+0000 7f10437fe640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f102c005940 tx=0x7f102c0043b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:53.935 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.932+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c041070 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.932+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f102c02ccc0 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.932+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f102c003e20 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.932+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10441983a0 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.932+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10441a46a0 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.933+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f104404f050 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.934+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f102c044070 con 0x7f1044100e00 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.934+0000 7f10417fa640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 0x7f10300a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.934+0000 7f1043fff640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 0x7f10300a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:53.936 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.934+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f102c0f0cd0 con 0x7f1044100e00 2024-03-09T00:16:53.937 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.935+0000 7f1043fff640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 0x7f10300a4ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1034005e10 tx=0x7f1034005da0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:53.938 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:53.939+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f102c0bd750 con 0x7f1044100e00 2024-03-09T00:16:54.136 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.137+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}) v1 -- 0x7f1044068970 con 0x7f10300a2a10 2024-03-09T00:16:54.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.145+0000 7f10417fa640 1 -- 172.21.15.73:0/765878069 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+350 (secure 0 0 0) 0x7f1044068970 con 0x7f10300a2a10 2024-03-09T00:16:54.144 INFO:teuthology.orchestra.run.smithi073.stdout:{"fs": "cephfs", "subvol": null, "group": null, "path": "/", "rel_path": "/", "schedule": "1m", "retention": {"m": 6, "h": 3}, "start": "2024-03-09T00:00:00", "created": "2024-03-09T00:06:51", "first": "2024-03-09T00:07:00", "last": "2024-03-09T00:16:00", "last_pruned": "2024-03-09T00:16:00", "created_count": 10, "pruned_count": 4, "active": true} 2024-03-09T00:16:54.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 msgr2=0x7f10300a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 0x7f10300a4ed0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f1034005e10 tx=0x7f1034005da0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.146 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 msgr2=0x7f104419dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 crypto rx=0x7f102c005940 tx=0x7f102c0043b0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 shutdown_connections 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1044105190 0x7f1044197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f10300a2a10 0x7f10300a4ed0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1044100e00 0x7f104419dd20 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 --2- 172.21.15.73:0/765878069 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1044100440 0x7f104419d7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 >> 172.21.15.73:0/765878069 conn(0x7f1044077040 msgr2=0x7f1044104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 shutdown_connections 2024-03-09T00:16:54.147 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.147+0000 7f104a560640 1 -- 172.21.15.73:0/765878069 wait complete. 2024-03-09T00:16:54.157 DEBUG:teuthology.orchestra.run.smithi073:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs snap-schedule list --fs=cephfs --path=/ --recursive=true' 2024-03-09T00:16:54.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 -- 172.21.15.73:0/3502055276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 msgr2=0x7f982810d020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3502055276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982810d020 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 crypto rx=0x7f9818009900 tx=0x7f981802bc30 comp rx=0 tx=0).stop 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 -- 172.21.15.73:0/3502055276 shutdown_connections 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3502055276 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982810d020 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3502055276 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 0x7f98281008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3502055276 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 0x7f9828105570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 -- 172.21.15.73:0/3502055276 >> 172.21.15.73:0/3502055276 conn(0x7f9828077040 msgr2=0x7f9828077450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 -- 172.21.15.73:0/3502055276 shutdown_connections 2024-03-09T00:16:54.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.261+0000 7f982eeb2640 1 -- 172.21.15.73:0/3502055276 wait complete. 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 Processor -- start 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 -- start start 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 0x7f982819d8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 0x7f9828197a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9828110730 con 0x7f9828100e00 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281105b0 con 0x7f9828105190 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982eeb2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98281108b0 con 0x7f9828100440 2024-03-09T00:16:54.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f9827fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f9827fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:42766/0 (socket says 172.21.15.73:42766) 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f9827fff640 1 -- 172.21.15.73:0/3935728044 learned_addr learned my addr 172.21.15.73:0/3935728044 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.262+0000 7f982cc4e640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 0x7f982819d8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f982d44f640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 0x7f9828197a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 -- 172.21.15.73:0/3935728044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 msgr2=0x7f982819d8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 0x7f982819d8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 -- 172.21.15.73:0/3935728044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 msgr2=0x7f9828197a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 0x7f9828197a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 -- 172.21.15.73:0/3935728044 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98281981f0 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9827fff640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f981000b4a0 tx=0x7f981000b960 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810014070 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9810010af0 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98281984e0 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9810017d10 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.263+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98281a4780 con 0x7f9828100e00 2024-03-09T00:16:54.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.264+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f982804f050 con 0x7f9828100e00 2024-03-09T00:16:54.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.265+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f9810017070 con 0x7f9828100e00 2024-03-09T00:16:54.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.266+0000 7f9825ffb640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 0x7f98140a4ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:16:54.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.266+0000 7f982cc4e640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 0x7f98140a4ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:16:54.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.266+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f98100ce4e0 con 0x7f9828100e00 2024-03-09T00:16:54.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.266+0000 7f982cc4e640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 0x7f98140a4ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9828198bb0 tx=0x7f982000d040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:16:54.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.270+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f981009af60 con 0x7f9828100e00 2024-03-09T00:16:54.466 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.467+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 --> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] -- mgr_command(tid 0: {"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9828068970 con 0x7f98140a2a10 2024-03-09T00:16:54.473 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.474+0000 7f9825ffb640 1 -- 172.21.15.73:0/3935728044 <== mgr.15564 v2:172.21.15.73:6800/2716694463 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+10 (secure 0 0 0) 0x7f9828068970 con 0x7f98140a2a10 2024-03-09T00:16:54.473 INFO:teuthology.orchestra.run.smithi073.stdout:/ 1m 6m3h 2024-03-09T00:16:54.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 msgr2=0x7f98140a4ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.475 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 0x7f98140a4ed0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f9828198bb0 tx=0x7f982000d040 comp rx=0 tx=0).stop 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 msgr2=0x7f982819de00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 crypto rx=0x7f981000b4a0 tx=0x7f981000b960 comp rx=0 tx=0).stop 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 shutdown_connections 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f9828105190 0x7f9828197a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f98140a2a10 0x7f98140a4ed0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f9828100e00 0x7f982819de00 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 --2- 172.21.15.73:0/3935728044 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f9828100440 0x7f982819d8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:16:54.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 >> 172.21.15.73:0/3935728044 conn(0x7f9828077040 msgr2=0x7f9828104000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:16:54.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 shutdown_connections 2024-03-09T00:16:54.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:16:54.476+0000 7f982eeb2640 1 -- 172.21.15.73:0/3935728044 wait complete. 2024-03-09T00:16:54.485 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:54 smithi073 ceph-mon[32309]: pgmap v312: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:54.485 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:54 smithi073 ceph-mon[32309]: scrub summary: idle 2024-03-09T00:16:54.486 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-09T00:16:54.497 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-09T00:16:54.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:54 smithi102 ceph-mon[36106]: pgmap v312: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:54.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:54 smithi102 ceph-mon[36106]: scrub summary: idle 2024-03-09T00:16:54.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:54 smithi179 ceph-mon[36637]: pgmap v312: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:16:54.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:54 smithi179 ceph-mon[36637]: scrub summary: idle 2024-03-09T00:16:55.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:55 smithi102 ceph-mon[36106]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:55.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:55 smithi179 ceph-mon[36637]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:55.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:55 smithi073 ceph-mon[32309]: from='client.17262 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule status", "fs": "cephfs", "path": "/", "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:56.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:56 smithi102 ceph-mon[36106]: from='client.17268 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:56.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:56 smithi102 ceph-mon[36106]: pgmap v313: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:56.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:56 smithi179 ceph-mon[36637]: from='client.17268 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:56.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:56 smithi179 ceph-mon[36637]: pgmap v313: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:56.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:56 smithi073 ceph-mon[32309]: from='client.17268 -' entity='client.admin' cmd=[{"prefix": "fs snap-schedule list", "fs": "cephfs", "path": "/", "recursive": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-09T00:16:56.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:56 smithi073 ceph-mon[32309]: pgmap v313: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:16:58.457 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:16:58 smithi073 ceph-mon[32309]: pgmap v314: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 0 B/s wr, 221 op/s 2024-03-09T00:16:58.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:16:58 smithi102 ceph-mon[36106]: pgmap v314: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 0 B/s wr, 221 op/s 2024-03-09T00:16:58.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:16:58 smithi179 ceph-mon[36637]: pgmap v314: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 220 KiB/s rd, 0 B/s wr, 221 op/s 2024-03-09T00:17:00.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:00 smithi102 ceph-mon[36106]: pgmap v315: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:00.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:00 smithi179 ceph-mon[36637]: pgmap v315: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:00.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:00 smithi073 ceph-mon[32309]: pgmap v315: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:02.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:02 smithi102 ceph-mon[36106]: pgmap v316: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:02.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:02 smithi179 ceph-mon[36637]: pgmap v316: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:02.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:02 smithi073 ceph-mon[32309]: pgmap v316: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 289 KiB/s rd, 0 B/s wr, 290 op/s 2024-03-09T00:17:03.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:03 smithi102 ceph-mon[36106]: pgmap v317: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:03.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:03 smithi179 ceph-mon[36637]: pgmap v317: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:03.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:03 smithi073 ceph-mon[32309]: pgmap v317: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:04.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:04 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:17:04.521 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:04 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:17:04.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:04 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:17:05.406 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:05 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.408 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:05 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.408 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:05 smithi073 ceph-mon[32309]: pgmap v318: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 44 KiB/s wr, 307 op/s 2024-03-09T00:17:05.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:05 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:05 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:05 smithi102 ceph-mon[36106]: pgmap v318: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 44 KiB/s wr, 307 op/s 2024-03-09T00:17:05.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:05 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:05 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:05 smithi179 ceph-mon[36637]: pgmap v318: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 299 KiB/s rd, 44 KiB/s wr, 307 op/s 2024-03-09T00:17:06.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:17:06.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:06 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:17:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:17:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:17:06.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:17:06.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:06 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:17:06.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi179 to 174.1M 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi179 to 182618419: error parsing value: Value '182618419' is below minimum 939524096 2024-03-09T00:17:06.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:06.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi102 to 174.1M 2024-03-09T00:17:06.838 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:06 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi102 to 182632755: error parsing value: Value '182632755' is below minimum 939524096 2024-03-09T00:17:07.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:17:07.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:07 smithi073 ceph-mon[32309]: pgmap v319: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:07.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:07 smithi102 ceph-mon[36106]: pgmap v319: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:07.835 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: Adjusting osd_memory_target on smithi073 to 176.2M 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: Unable to set osd_memory_target on smithi073 to 184762368: error parsing value: Value '184762368' is below minimum 939524096 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:17:07.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:17:07.837 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:07 smithi179 ceph-mon[36637]: pgmap v319: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 300 KiB/s rd, 44 KiB/s wr, 308 op/s 2024-03-09T00:17:10.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:10 smithi102 ceph-mon[36106]: osdmap e103: 12 total, 12 up, 12 in 2024-03-09T00:17:10.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:10 smithi102 ceph-mon[36106]: pgmap v321: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:10.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:10 smithi179 ceph-mon[36637]: osdmap e103: 12 total, 12 up, 12 in 2024-03-09T00:17:10.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:10 smithi179 ceph-mon[36637]: pgmap v321: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:10.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:10 smithi073 ceph-mon[32309]: osdmap e103: 12 total, 12 up, 12 in 2024-03-09T00:17:10.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:10 smithi073 ceph-mon[32309]: pgmap v321: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:11.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:11 smithi102 ceph-mon[36106]: pgmap v322: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:11.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:11 smithi179 ceph-mon[36637]: pgmap v322: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:11.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:11 smithi073 ceph-mon[32309]: pgmap v322: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:17:13.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:13 smithi102 ceph-mon[36106]: pgmap v323: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:13.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:13 smithi179 ceph-mon[36637]: pgmap v323: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:13.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:13 smithi073 ceph-mon[32309]: pgmap v323: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:16.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:15 smithi102 ceph-mon[36106]: osdmap e104: 12 total, 12 up, 12 in 2024-03-09T00:17:16.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:15 smithi102 ceph-mon[36106]: pgmap v325: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:17:16.335 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:15 smithi179 ceph-mon[36637]: osdmap e104: 12 total, 12 up, 12 in 2024-03-09T00:17:16.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:15 smithi179 ceph-mon[36637]: pgmap v325: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:17:16.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:15 smithi073 ceph-mon[32309]: osdmap e104: 12 total, 12 up, 12 in 2024-03-09T00:17:16.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:15 smithi073 ceph-mon[32309]: pgmap v325: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 767 B/s rd, 127 B/s wr, 1 op/s 2024-03-09T00:17:17.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:17 smithi102 ceph-mon[36106]: pgmap v326: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 251 B/s wr, 2 op/s 2024-03-09T00:17:17.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:17 smithi179 ceph-mon[36637]: pgmap v326: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 251 B/s wr, 2 op/s 2024-03-09T00:17:17.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:17 smithi073 ceph-mon[32309]: pgmap v326: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.5 KiB/s rd, 251 B/s wr, 2 op/s 2024-03-09T00:17:19.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:19 smithi102 ceph-mon[36106]: pgmap v327: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:19.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:19 smithi179 ceph-mon[36637]: pgmap v327: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:19.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:19 smithi073 ceph-mon[32309]: pgmap v327: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:21.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:21 smithi102 ceph-mon[36106]: pgmap v328: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:21.574 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:21 smithi179 ceph-mon[36637]: pgmap v328: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:21.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:21 smithi073 ceph-mon[32309]: pgmap v328: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 204 B/s wr, 2 op/s 2024-03-09T00:17:22.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-09T00:17:22.618 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-09T00:17:22.738 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- 172.21.15.73:0/613265865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 msgr2=0x7fb40c1018d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 --2- 172.21.15.73:0/613265865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c1018d0 secure :-1 s=READY pgs=1020 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc009900 tx=0x7fb3fc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- 172.21.15.73:0/613265865 shutdown_connections 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 --2- 172.21.15.73:0/613265865 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c1065c0 0x7fb40c10e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 --2- 172.21.15.73:0/613265865 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c101ea0 0x7fb40c105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 --2- 172.21.15.73:0/613265865 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c1018d0 unknown :-1 s=CLOSED pgs=1020 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- 172.21.15.73:0/613265865 >> 172.21.15.73:0/613265865 conn(0x7fb40c0786f0 msgr2=0x7fb40c078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- 172.21.15.73:0/613265865 shutdown_connections 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- 172.21.15.73:0/613265865 wait complete. 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 Processor -- start 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.737+0000 7fb412659640 1 -- start start 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 0x7fb40c19dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c197950 0x7fb40c197e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c111670 con 0x7fb40c1014f0 2024-03-09T00:17:22.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1114f0 con 0x7fb40c197950 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb412659640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1117f0 con 0x7fb40c1065c0 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39192/0 (socket says 172.21.15.73:39192) 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 -- 172.21.15.73:0/4253969308 learned_addr learned my addr 172.21.15.73:0/4253969308 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb410bf6640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c197950 0x7fb40c197e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 -- 172.21.15.73:0/4253969308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 msgr2=0x7fb40c19dce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 0x7fb40c19dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.741 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 0x7fb40c19dce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 -- 172.21.15.73:0/4253969308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c197950 msgr2=0x7fb40c197e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c197950 0x7fb40c197e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 -- 172.21.15.73:0/4253969308 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb40c198470 con 0x7fb40c1014f0 2024-03-09T00:17:22.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 0x7fb40c19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:22.742 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb40bfff640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc006010 tx=0x7fb3fc02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.738+0000 7fb4097fa640 1 -- 172.21.15.73:0/4253969308 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc041070 con 0x7fb40c1014f0 2024-03-09T00:17:22.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.739+0000 7fb4097fa640 1 -- 172.21.15.73:0/4253969308 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3fc004450 con 0x7fb40c1014f0 2024-03-09T00:17:22.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.739+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb40c198700 con 0x7fb40c1014f0 2024-03-09T00:17:22.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.739+0000 7fb4097fa640 1 -- 172.21.15.73:0/4253969308 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc034440 con 0x7fb40c1014f0 2024-03-09T00:17:22.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.739+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb40c1a48f0 con 0x7fb40c1014f0 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb4097fa640 1 -- 172.21.15.73:0/4253969308 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fb3fc044070 con 0x7fb40c1014f0 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 Processor -- start 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 -- start start 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb4097fa640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 0x7fb3f40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775abc520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.750 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.751 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775aca070 0x561775acc460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.741+0000 7fb415238740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x561775a1fe00 con 0x561775abcaf0 2024-03-09T00:17:22.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb415238740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617759c8b10 con 0x561775aca070 2024-03-09T00:17:22.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb415238740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x561775a1bea0 con 0x561775abc140 2024-03-09T00:17:22.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb410bf6640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 0x7fb3f40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.754 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb4097fa640 1 -- 172.21.15.73:0/4253969308 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb3fc0f0dc0 con 0x7fb40c1014f0 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39204/0 (socket says 172.21.15.73:39204) 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3699328452 learned_addr learned my addr 172.21.15.73:0/3699328452 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3699328452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 msgr2=0x561775abc520 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775abc520 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3699328452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775aca070 msgr2=0x561775acc460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775aca070 0x561775acc460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.755 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3699328452 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5617757fe7d0 con 0x561775abcaf0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb410bf6640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 0x7fb3f40a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb404007de0 tx=0x7fb40400f790 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb400009bc0 tx=0x7fb40000e6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6d33fd6e2bb8440e server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb3f27fc640 1 -- 172.21.15.73:0/3699328452 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400016070 con 0x561775abcaf0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb3f27fc640 1 -- 172.21.15.73:0/3699328452 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb400011950 con 0x561775abcaf0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.742+0000 7fb3f27fc640 1 -- 172.21.15.73:0/3699328452 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400010830 con 0x561775abcaf0 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- 172.21.15.73:0/3699328452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 msgr2=0x561775ac9b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.756 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 secure :-1 s=READY pgs=1022 cs=0 l=1 rev1=1 crypto rx=0x7fb400009bc0 tx=0x7fb40000e6c0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- 172.21.15.73:0/3699328452 shutdown_connections 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775aca070 0x561775acc460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac9b30 unknown :-1 s=CLOSED pgs=1022 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 --2- 172.21.15.73:0/3699328452 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775abc520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- 172.21.15.73:0/3699328452 >> 172.21.15.73:0/3699328452 conn(0x561775ab6e90 msgr2=0x561775ab7270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- 172.21.15.73:0/3699328452 shutdown_connections 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- 172.21.15.73:0/3699328452 wait complete. 2024-03-09T00:17:22.757 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.743+0000 7fb415238740 1 -- start start 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 Processor -- start 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775ac98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ac6c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x561775acce30 con 0x561775aca070 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x5617759dc0a0 con 0x561775abcaf0 2024-03-09T00:17:22.758 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x561775accfb0 con 0x561775abc140 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb415238740 10 client.0 _ll_register_callbacks cb 0x5617759dba70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39216/0 (socket says 172.21.15.73:39216) 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 -- 172.21.15.73:0/1217741277 learned_addr learned my addr 172.21.15.73:0/1217741277 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ac6c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 -- 172.21.15.73:0/1217741277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 msgr2=0x561775ac98b0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775ac98b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 -- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 msgr2=0x561775ac6c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.759 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ac6c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 -- 172.21.15.73:0/1217741277 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5617757343f0 con 0x561775aca070 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb410bf6640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb404010db0 tx=0x7fb404004ab0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=15938de93e45d9c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.744+0000 7fb3eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb40401f650 con 0x561775aca070 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb400011950 con 0x561775aca070 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb40401f7f0 con 0x561775aca070 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb40402d050 con 0x561775aca070 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 10 client.17280 fetch_fsmap learned FSMap version 20 2024-03-09T00:17:22.760 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x5617756f40f0 con 0x561775aca070 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb400011950 con 0x561775aca070 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb404028900 con 0x561775aca070 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 10 client.17280 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 10 client.17280 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 20 client.17280 populate_metadata read hostname 'smithi073' 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775ad9cf0 0x561775adc130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 4 client.17280 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:17:22.761 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561775a2d340 con 0x561775ad9cf0 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.745+0000 7fb40bfff640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775ad9cf0 0x561775adc130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.746+0000 7fb3eb7fe640 10 client.17280 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.746+0000 7fb40bfff640 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775ad9cf0 0x561775adc130 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a538b924c925e8b7 server_cookie=5e721426ad4b2fc0 in_seq=0 out_seq=0 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.746+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/1217741277 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x561775a2d340 con 0x561775ad9cf0 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.746+0000 7fb3eb7fe640 10 client.17280 handle_command_reply: tid=0 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.746+0000 7fb415238740 1 client.17280 shutdown 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 msgr2=0x561775ac7180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.762 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 crypto rx=0x7fb404010db0 tx=0x7fb404004ab0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 shutdown_connections 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775ad9cf0 0x561775adc130 crc :-1 s=READY pgs=209 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775aca070 0x561775ac7180 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ac6c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb3eb7fe640 0 client.17280 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 --2- 172.21.15.73:0/1217741277 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775abc140 0x561775ac98b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 >> 172.21.15.73:0/1217741277 conn(0x561775ab6e90 msgr2=0x561775ab8be0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.763 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 shutdown_connections 2024-03-09T00:17:22.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 1 -- 172.21.15.73:0/1217741277 wait complete. 2024-03-09T00:17:22.764 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.747+0000 7fb415238740 20 client.17280 trim_cache size 0 max 16384 2024-03-09T00:17:22.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.761+0000 7fb415238740 1 Processor -- start 2024-03-09T00:17:22.770 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 -- start start 2024-03-09T00:17:22.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abc140 0x561775ac9370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775ad1260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1aa520 con 0x561775abcaf0 2024-03-09T00:17:22.771 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1acac0 con 0x561775abc140 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb415238740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1aab20 con 0x561775aca070 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39220/0 (socket says 172.21.15.73:39220) 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3010177190 learned_addr learned my addr 172.21.15.73:0/3010177190 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40bfff640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abc140 0x561775ac9370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb410bf6640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775ad1260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.772 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3010177190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 msgr2=0x561775ad1260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775ad1260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3010177190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abc140 msgr2=0x561775ac9370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abc140 0x561775ac9370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 -- 172.21.15.73:0/3010177190 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561775abc890 con 0x561775abcaf0 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb410bf6640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775ad1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb400009bc0 tx=0x7fb400009940 comp rx=0 tx=0).ready entity=mon.0 client_cookie=78433fed2d28c6c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.762+0000 7fb3f1ffb640 1 -- 172.21.15.73:0/3010177190 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400029070 con 0x561775abcaf0 2024-03-09T00:17:22.773 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb3f1ffb640 1 -- 172.21.15.73:0/3010177190 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb400009e70 con 0x561775abcaf0 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb3f1ffb640 1 -- 172.21.15.73:0/3010177190 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb400016040 con 0x561775abcaf0 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 -- 172.21.15.73:0/3010177190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 msgr2=0x561775ac98b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 crypto rx=0x7fb400009bc0 tx=0x7fb400009940 comp rx=0 tx=0).stop 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 -- 172.21.15.73:0/3010177190 shutdown_connections 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775ad1260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abcaf0 0x561775ac98b0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 --2- 172.21.15.73:0/3010177190 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abc140 0x561775ac9370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 -- 172.21.15.73:0/3010177190 >> 172.21.15.73:0/3010177190 conn(0x561775ab6e90 msgr2=0x561775ac6c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 -- 172.21.15.73:0/3010177190 shutdown_connections 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.763+0000 7fb415238740 1 -- 172.21.15.73:0/3010177190 wait complete. 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 -- start start 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 Processor -- start 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ad1750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.775 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775acf810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1ab140 con 0x561775abc140 2024-03-09T00:17:22.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1ac460 con 0x561775abcaf0 2024-03-09T00:17:22.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40c1ab7a0 con 0x561775aca070 2024-03-09T00:17:22.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb415238740 10 client.0 _ll_register_callbacks cb 0x7fb404007ed0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:17:22.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39236/0 (socket says 172.21.15.73:39236) 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 -- 172.21.15.73:0/2105517147 learned_addr learned my addr 172.21.15.73:0/2105517147 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ad1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb410bf6640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775acf810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 -- 172.21.15.73:0/2105517147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 msgr2=0x561775acf810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775acf810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 -- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 msgr2=0x561775ad1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ad1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.764+0000 7fb40bfff640 1 -- 172.21.15.73:0/2105517147 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561775ad0010 con 0x561775abc140 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb40bfff640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005940 tx=0x7fb3fc0bcb60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ddaa94b8dd6d1c57 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3fc04b040 con 0x561775abc140 2024-03-09T00:17:22.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fb3fc0bb1e0 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb400009e70 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3fc0c1290 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 10 client.17286 fetch_fsmap learned FSMap version 20 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x561775b30db0 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb400009e70 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fb3fc0035f0 con 0x561775abc140 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 10 client.17286 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 10 client.17286 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:17:22.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 20 client.17286 populate_metadata read hostname 'smithi073' 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775b31430 0x561775b33850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 4 client.17286 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x561775a27690 con 0x561775b31430 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.765+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775b31430 0x561775b33850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb3eb7fe640 10 client.17286 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb40b7fe640 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775b31430 0x561775b33850 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d34d40c31c7bea99 server_cookie=7481d5979dd6040b in_seq=0 out_seq=0 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb3eb7fe640 1 -- 172.21.15.73:0/2105517147 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x561775a27690 con 0x561775b31430 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb3eb7fe640 10 client.17286 handle_command_reply: tid=0 2024-03-09T00:17:22.780 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 client.17286 shutdown 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 msgr2=0x561775ad1210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc005940 tx=0x7fb3fc0bcb60 comp rx=0 tx=0).stop 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 shutdown_connections 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x561775b31430 0x561775b33850 crc :-1 s=READY pgs=210 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x561775aca070 0x561775acf810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x561775abcaf0 0x561775ad1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb3eb7fe640 0 client.17286 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:22.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 --2- 172.21.15.73:0/2105517147 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x561775abc140 0x561775ad1210 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 >> 172.21.15.73:0/2105517147 conn(0x561775ab6e90 msgr2=0x561775ac1ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 shutdown_connections 2024-03-09T00:17:22.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 1 -- 172.21.15.73:0/2105517147 wait complete. 2024-03-09T00:17:22.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.766+0000 7fb415238740 20 client.17286 trim_cache size 0 max 16384 2024-03-09T00:17:22.782 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:22.783 INFO:teuthology.orchestra.run.smithi073.stdout: "status": "no active scrubs running", 2024-03-09T00:17:22.783 INFO:teuthology.orchestra.run.smithi073.stdout: "scrubs": {} 2024-03-09T00:17:22.783 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:22.784 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 msgr2=0x7fb3f40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 0x7fb3f40a4e30 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fb404007de0 tx=0x7fb40400f790 comp rx=0 tx=0).stop 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 msgr2=0x7fb40c19d7a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 crypto rx=0x7fb3fc006010 tx=0x7fb3fc02cc70 comp rx=0 tx=0).stop 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 shutdown_connections 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fb40c197950 0x7fb40c197e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fb40c1065c0 0x7fb40c19dce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fb3f40a2970 0x7fb3f40a4e30 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.785 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 --2- 172.21.15.73:0/4253969308 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fb40c1014f0 0x7fb40c19d7a0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 >> 172.21.15.73:0/4253969308 conn(0x7fb40c0786f0 msgr2=0x7fb40c0ff380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 shutdown_connections 2024-03-09T00:17:22.786 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.767+0000 7fb412659640 1 -- 172.21.15.73:0/4253969308 wait complete. 2024-03-09T00:17:22.786 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-09T00:17:22.787 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:17:22.890 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 -- 172.21.15.73:0/4224017079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030100e90 msgr2=0x7f7030104e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 --2- 172.21.15.73:0/4224017079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030100e90 0x7f7030104e80 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 crypto rx=0x7f7020009900 tx=0x7f702002bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 -- 172.21.15.73:0/4224017079 shutdown_connections 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 --2- 172.21.15.73:0/4224017079 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f7030105560 0x7f703010d130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 --2- 172.21.15.73:0/4224017079 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030100e90 0x7f7030104e80 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 --2- 172.21.15.73:0/4224017079 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f70301004e0 0x7f70301008c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 -- 172.21.15.73:0/4224017079 >> 172.21.15.73:0/4224017079 conn(0x7f7030077350 msgr2=0x7f7030077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:22.891 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 -- 172.21.15.73:0/4224017079 shutdown_connections 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.889+0000 7f703863d640 1 -- 172.21.15.73:0/4224017079 wait complete. 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 Processor -- start 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 -- start start 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70301004e0 0x7f703019c750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7030196950 0x7f7030196e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70301105d0 con 0x7f7030105560 2024-03-09T00:17:22.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7030110450 con 0x7f70301004e0 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f703863d640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7030110750 con 0x7f7030196950 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f7035bd8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.890+0000 7f7035bd8640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39252/0 (socket says 172.21.15.73:39252) 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 -- 172.21.15.73:0/3486352907 learned_addr learned my addr 172.21.15.73:0/3486352907 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f70363d9640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70301004e0 0x7f703019c750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 -- 172.21.15.73:0/3486352907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7030196950 msgr2=0x7f7030196e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7030196950 0x7f7030196e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 -- 172.21.15.73:0/3486352907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70301004e0 msgr2=0x7f703019c750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:22.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70301004e0 0x7f703019c750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 -- 172.21.15.73:0/3486352907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030197470 con 0x7f7030105560 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f7035bd8640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7020000c00 tx=0x7f702002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020041070 con 0x7f7030105560 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7020004450 con 0x7f7030105560 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7030197700 con 0x7f7030105560 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7020034440 con 0x7f7030105560 2024-03-09T00:17:22.894 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.891+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70301a38a0 con 0x7f7030105560 2024-03-09T00:17:22.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.892+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7030068530 con 0x7f7030105560 2024-03-09T00:17:22.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.893+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f7020044070 con 0x7f7030105560 2024-03-09T00:17:22.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.894+0000 7f70277fe640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 0x7f701c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:22.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.894+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f70200f0e70 con 0x7f7030105560 2024-03-09T00:17:22.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.894+0000 7f70363d9640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 0x7f701c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:22.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.894+0000 7f70363d9640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 0x7f701c0a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f702c0058c0 tx=0x7f702c00a2d0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:22.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:22.898+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f70200bd8f0 con 0x7f7030105560 2024-03-09T00:17:23.109 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.111+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7030100e90 con 0x7f7030105560 2024-03-09T00:17:23.110 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.111+0000 7f70277fe640 1 -- 172.21.15.73:0/3486352907 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f70200bd1f0 con 0x7f7030105560 2024-03-09T00:17:23.111 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:17:23.111 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:17:23.111 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:17:23.112 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 msgr2=0x7f701c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 0x7f701c0a4e30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f702c0058c0 tx=0x7f702c00a2d0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 msgr2=0x7f703019cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 crypto rx=0x7f7020000c00 tx=0x7f702002cc70 comp rx=0 tx=0).stop 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 shutdown_connections 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f7030196950 0x7f7030196e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f701c0a2970 0x7f701c0a4e30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f7030105560 0x7f703019cc90 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.113 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 --2- 172.21.15.73:0/3486352907 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f70301004e0 0x7f703019c750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 >> 172.21.15.73:0/3486352907 conn(0x7f7030077350 msgr2=0x7f70300fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 shutdown_connections 2024-03-09T00:17:23.114 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.113+0000 7f703863d640 1 -- 172.21.15.73:0/3486352907 wait complete. 2024-03-09T00:17:23.123 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-09T00:17:23.123 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-09T00:17:23.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 -- 172.21.15.73:0/2104646138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801065a0 msgr2=0x7fd78010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 --2- 172.21.15.73:0/2104646138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801065a0 0x7fd78010e170 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 crypto rx=0x7fd778009ca0 tx=0x7fd77802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 -- 172.21.15.73:0/2104646138 shutdown_connections 2024-03-09T00:17:23.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 --2- 172.21.15.73:0/2104646138 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801065a0 0x7fd78010e170 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.231 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 --2- 172.21.15.73:0/2104646138 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd780101e80 0x7fd780105ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 --2- 172.21.15.73:0/2104646138 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd7801018b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 -- 172.21.15.73:0/2104646138 >> 172.21.15.73:0/2104646138 conn(0x7fd7800fcd90 msgr2=0x7fd7800ff1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 -- 172.21.15.73:0/2104646138 shutdown_connections 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.229+0000 7fd78530a640 1 -- 172.21.15.73:0/2104646138 wait complete. 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 Processor -- start 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 -- start start 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd78019d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.232 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 0x7fd78019dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801116b0 con 0x7fd7801979d0 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd780111530 con 0x7fd7801065a0 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd78530a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd780111830 con 0x7fd7801014d0 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39266/0 (socket says 172.21.15.73:39266) 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 -- 172.21.15.73:0/2025520326 learned_addr learned my addr 172.21.15.73:0/2025520326 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:23.233 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd7767fc640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 0x7fd78019dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 -- 172.21.15.73:0/2025520326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 msgr2=0x7fd78019d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77effd640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd78019d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd78019d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 -- 172.21.15.73:0/2025520326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 msgr2=0x7fd78019dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 0x7fd78019dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77f7fe640 1 -- 172.21.15.73:0/2025520326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7801984f0 con 0x7fd7801979d0 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd7767fc640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 0x7fd78019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:23.234 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.230+0000 7fd77effd640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd78019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd77802c700 tx=0x7fd778032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd77cff9640 1 -- 172.21.15.73:0/2025520326 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd778039040 con 0x7fd7801979d0 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd780198780 con 0x7fd7801979d0 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd77cff9640 1 -- 172.21.15.73:0/2025520326 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd778044070 con 0x7fd7801979d0 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd77cff9640 1 -- 172.21.15.73:0/2025520326 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780042a0 con 0x7fd7801979d0 2024-03-09T00:17:23.235 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.231+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7801a4970 con 0x7fd7801979d0 2024-03-09T00:17:23.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd77cff9640 1 -- 172.21.15.73:0/2025520326 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd778004440 con 0x7fd7801979d0 2024-03-09T00:17:23.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd787fc0740 1 Processor -- start 2024-03-09T00:17:23.242 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd787fc0740 1 -- start start 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd77cff9640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 0x7fd76c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff7e50 0x559557ff8230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.233+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559558005ab0 0x559558007ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd787fc0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559557fbf360 con 0x559557ff8770 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd787fc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559557f5bec0 con 0x559557ff7e50 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd787fc0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559557f04d30 con 0x559558005ab0 2024-03-09T00:17:23.243 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 0x7fd76c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd77cff9640 1 -- 172.21.15.73:0/2025520326 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fd7780f15b0 con 0x7fd7801979d0 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39276/0 (socket says 172.21.15.73:39276) 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 -- 172.21.15.73:0/3958856443 learned_addr learned my addr 172.21.15.73:0/3958856443 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559558005ab0 0x559558007ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd77effd640 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff7e50 0x559557ff8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 -- 172.21.15.73:0/3958856443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559558005ab0 msgr2=0x559558007ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559558005ab0 0x559558007ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.244 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 -- 172.21.15.73:0/3958856443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff7e50 msgr2=0x559557ff8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff7e50 0x559557ff8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 -- 172.21.15.73:0/3958856443 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559557f03bc0 con 0x559557ff8770 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 0x7fd76c0a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd76800dfd0 tx=0x7fd76800d420 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd7680058c0 tx=0x7fd76801a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9719d2b1e44a63fd server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7757fa640 1 -- 172.21.15.73:0/3958856443 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768023070 con 0x559557ff8770 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7757fa640 1 -- 172.21.15.73:0/3958856443 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd768005500 con 0x559557ff8770 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.234+0000 7fd7757fa640 1 -- 172.21.15.73:0/3958856443 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768004aa0 con 0x559557ff8770 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 -- 172.21.15.73:0/3958856443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 msgr2=0x559558005570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.245 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 secure :-1 s=READY pgs=1030 cs=0 l=1 rev1=1 crypto rx=0x7fd7680058c0 tx=0x7fd76801a000 comp rx=0 tx=0).stop 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 -- 172.21.15.73:0/3958856443 shutdown_connections 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559558005ab0 0x559558007ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558005570 unknown :-1 s=CLOSED pgs=1030 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3958856443 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff7e50 0x559557ff8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 -- 172.21.15.73:0/3958856443 >> 172.21.15.73:0/3958856443 conn(0x559557ff28d0 msgr2=0x559557ff2cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 -- 172.21.15.73:0/3958856443 shutdown_connections 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.235+0000 7fd787fc0740 1 -- 172.21.15.73:0/3958856443 wait complete. 2024-03-09T00:17:23.246 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 -- start start 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 Processor -- start 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558002a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x5595580034e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x559557efa4a0 con 0x559557ff8770 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x559557f57f60 con 0x559558005ab0 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x559558008890 con 0x559557ff7e50 2024-03-09T00:17:23.247 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd787fc0740 10 client.0 _ll_register_callbacks cb 0x559557f55d60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39278/0 (socket says 172.21.15.73:39278) 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 -- 172.21.15.73:0/245837284 learned_addr learned my addr 172.21.15.73:0/245837284 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x5595580034e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 -- 172.21.15.73:0/245837284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 msgr2=0x559558002a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558002a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 -- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 msgr2=0x5595580034e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.248 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x5595580034e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 -- 172.21.15.73:0/245837284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559557c703f0 con 0x559557ff8770 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7767fc640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd76800b950 tx=0x7fd76800b980 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34f1ea5c98dbcdd8 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7667fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.236+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd768034060 con 0x559557ff8770 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fd76801f440 con 0x559557ff8770 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd768005500 con 0x559557ff8770 2024-03-09T00:17:23.249 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd76803b050 con 0x559557ff8770 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 10 client.17304 fetch_fsmap learned FSMap version 20 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x559557d3a7d0 con 0x559557ff8770 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd768005500 con 0x559557ff8770 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd768027420 con 0x559557ff8770 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 10 client.17304 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 10 client.17304 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:17:23.250 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 20 client.17304 populate_metadata read hostname 'smithi073' 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559558015cb0 0x5595580180a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 4 client.17304 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559557f69400 con 0x559558015cb0 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.237+0000 7fd77effd640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559558015cb0 0x5595580180a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.238+0000 7fd7667fc640 10 client.17304 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.238+0000 7fd77effd640 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559558015cb0 0x5595580180a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=af80f3c238057d4a server_cookie=46b016af0a836ab1 in_seq=0 out_seq=0 2024-03-09T00:17:23.251 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd7667fc640 1 -- 172.21.15.73:0/245837284 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+19073 (crc 0 0 0) 0x559557f69400 con 0x559558015cb0 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd7667fc640 10 client.17304 handle_command_reply: tid=0 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 client.17304 shutdown 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 msgr2=0x559558002fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 secure :-1 s=READY pgs=1031 cs=0 l=1 rev1=1 crypto rx=0x7fd76800b950 tx=0x7fd76800b980 comp rx=0 tx=0).stop 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 shutdown_connections 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x559558015cb0 0x5595580180a0 crc :-1 s=READY pgs=211 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x5595580034e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.252 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd7667fc640 0 client.17304 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff8770 0x559558002fa0 unknown :-1 s=CLOSED pgs=1031 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 --2- 172.21.15.73:0/245837284 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558002a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 >> 172.21.15.73:0/245837284 conn(0x559557ff28d0 msgr2=0x559557ff4620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 shutdown_connections 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 1 -- 172.21.15.73:0/245837284 wait complete. 2024-03-09T00:17:23.253 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.239+0000 7fd787fc0740 20 client.17304 trim_cache size 0 max 16384 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.250+0000 7fd787fc0740 1 Processor -- start 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.250+0000 7fd787fc0740 1 -- start start 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558005190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff8770 0x559557fffce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.260 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801aa580 con 0x559558005ab0 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801acb20 con 0x559557ff8770 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd787fc0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801aab80 con 0x559557ff7e50 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39294/0 (socket says 172.21.15.73:39294) 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 -- 172.21.15.73:0/3593086415 learned_addr learned my addr 172.21.15.73:0/3593086415 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff8770 0x559557fffce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 -- 172.21.15.73:0/3593086415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 msgr2=0x559558005190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.261 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558005190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 -- 172.21.15.73:0/3593086415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff8770 msgr2=0x559557fffce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff8770 0x559557fffce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 -- 172.21.15.73:0/3593086415 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559557ff2400 con 0x559558005ab0 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd7780bd630 tx=0x7fd7780a8040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b13f0187fa01cc50 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd774ff9640 1 -- 172.21.15.73:0/3593086415 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780d6bd0 con 0x559558005ab0 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd774ff9640 1 -- 172.21.15.73:0/3593086415 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7780333a0 con 0x559558005ab0 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.251+0000 7fd774ff9640 1 -- 172.21.15.73:0/3593086415 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd7780d9160 con 0x559558005ab0 2024-03-09T00:17:23.262 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- 172.21.15.73:0/3593086415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 msgr2=0x559558000220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 crypto rx=0x7fd7780bd630 tx=0x7fd7780a8040 comp rx=0 tx=0).stop 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- 172.21.15.73:0/3593086415 shutdown_connections 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559558005ab0 0x559558000220 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559557ff8770 0x559557fffce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3593086415 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff7e50 0x559558005190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- 172.21.15.73:0/3593086415 >> 172.21.15.73:0/3593086415 conn(0x559557ff28d0 msgr2=0x559558001330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- 172.21.15.73:0/3593086415 shutdown_connections 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- 172.21.15.73:0/3593086415 wait complete. 2024-03-09T00:17:23.263 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 -- start start 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 Processor -- start 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.252+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff8770 0x55955800c1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x55955806c080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801ab1a0 con 0x559557ff7e50 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801ac4c0 con 0x559558005ab0 2024-03-09T00:17:23.264 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7801ab800 con 0x559557ff8770 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 10 client.0 _ll_register_callbacks cb 0x7fd768013420 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39302/0 (socket says 172.21.15.73:39302) 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 -- 172.21.15.73:0/3952673615 learned_addr learned my addr 172.21.15.73:0/3952673615 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77f7fe640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x55955806c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff8770 0x55955800c1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 -- 172.21.15.73:0/3952673615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff8770 msgr2=0x55955800c1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff8770 0x55955800c1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.265 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 -- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 msgr2=0x55955806c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x55955806c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 -- 172.21.15.73:0/3952673615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55955806c5c0 con 0x559557ff7e50 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd77effd640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x559558018870 tx=0x7fd77000aeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f166aa0150afb2b0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7667fc640 10 client.0 ms_handle_connect on v2:172.21.15.73:3300/0 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd770019e60 con 0x559557ff7e50 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mon.0 v2:172.21.15.73:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fd7700196e0 con 0x559557ff7e50 2024-03-09T00:17:23.266 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd7780333a0 con 0x559557ff7e50 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd77002a050 con 0x559557ff7e50 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 10 client.17310 fetch_fsmap learned FSMap version 20 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=4+}) v3 -- 0x55955806d900 con 0x559557ff7e50 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.253+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mon.0 v2:172.21.15.73:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd7780333a0 con 0x559557ff7e50 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mon.0 v2:172.21.15.73:3300/0 5 ==== fsmap(e 20) v1 ==== 7890+0+0 (secure 0 0 0) 0x7fd7700154c0 con 0x559557ff7e50 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 10 client.17310 fetch_fsmap finished waiting for FSMap version 20 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 10 client.17310 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-09T00:17:23.267 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 20 client.17310 populate_metadata read hostname 'smithi073' 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55955806e080 0x5595580704a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 4 client.17310 mds_command: new command op to 24515 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 --> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x559557f63750 con 0x55955806e080 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55955806e080 0x5595580704a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd7667fc640 10 client.17310 ms_handle_connect on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd7767fc640 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55955806e080 0x5595580704a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eb5e645c56a81bce server_cookie=872a8429d3ef50a in_seq=0 out_seq=0 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.254+0000 7fd7667fc640 1 -- 172.21.15.73:0/3952673615 <== mds.0 v2:172.21.15.102:6838/3264474111 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x559557f63750 con 0x55955806e080 2024-03-09T00:17:23.268 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd7667fc640 10 client.17310 handle_command_reply: tid=0 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 client.17310 shutdown 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 msgr2=0x55955800bc80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 crypto rx=0x559558018870 tx=0x7fd77000aeb0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 shutdown_connections 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:6838/3264474111,v1:172.21.15.102:6839/3264474111] conn(0x55955806e080 0x5595580704a0 crc :-1 s=READY pgs=212 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x559558005ab0 0x55955806c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd7667fc640 0 client.17310 ms_handle_reset on v2:172.21.15.102:6838/3264474111 2024-03-09T00:17:23.269 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x559557ff8770 0x55955800c1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 --2- 172.21.15.73:0/3952673615 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x559557ff7e50 0x55955800bc80 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 >> 172.21.15.73:0/3952673615 conn(0x559557ff28d0 msgr2=0x559558002a00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 shutdown_connections 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 1 -- 172.21.15.73:0/3952673615 wait complete. 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd787fc0740 20 client.17310 trim_cache size 0 max 16384 2024-03-09T00:17:23.270 INFO:teuthology.orchestra.run.smithi073.stdout:[] 2024-03-09T00:17:23.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 msgr2=0x7fd76c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 0x7fd76c0a4e30 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fd76800dfd0 tx=0x7fd76800d420 comp rx=0 tx=0).stop 2024-03-09T00:17:23.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 msgr2=0x7fd780197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:23.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 crypto rx=0x7fd77802c700 tx=0x7fd778032f20 comp rx=0 tx=0).stop 2024-03-09T00:17:23.272 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 shutdown_connections 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd76c0a2970 0x7fd76c0a4e30 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd7801979d0 0x7fd780197e80 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd7801065a0 0x7fd78019dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 --2- 172.21.15.73:0/2025520326 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd7801014d0 0x7fd78019d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 >> 172.21.15.73:0/2025520326 conn(0x7fd7800fcd90 msgr2=0x7fd7800ff0c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 shutdown_connections 2024-03-09T00:17:23.273 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:23.255+0000 7fd78530a640 1 -- 172.21.15.73:0/2025520326 wait complete. 2024-03-09T00:17:23.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:23 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3486352907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:17:23.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:23 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3486352907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:17:23.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:23 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3486352907' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:17:24.274 INFO:tasks.fwd_scrub.fs.[cephfs]:end scrubbing fs: cephfs 2024-03-09T00:17:24.274 INFO:tasks.fwd_scrub:done joining 2024-03-09T00:17:24.275 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-09T00:17:24.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:17:24.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- 172.21.15.73:0/23229584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100e10 msgr2=0x7f1db8104e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/23229584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100e10 0x7f1db8104e50 secure :-1 s=READY pgs=1034 cs=0 l=1 rev1=1 crypto rx=0x7f1da8009900 tx=0x7f1da802bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:24.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- 172.21.15.73:0/23229584 shutdown_connections 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/23229584 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8105530 0x7f1db810d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/23229584 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100e10 0x7f1db8104e50 unknown :-1 s=CLOSED pgs=1034 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/23229584 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8100460 0x7f1db8100840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- 172.21.15.73:0/23229584 >> 172.21.15.73:0/23229584 conn(0x7f1db80772f0 msgr2=0x7f1db8077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- 172.21.15.73:0/23229584 shutdown_connections 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- 172.21.15.73:0/23229584 wait complete. 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 Processor -- start 2024-03-09T00:17:24.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- start start 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8105530 0x7f1db8198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 0x7f1db81a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db81105f0 con 0x7f1db8100460 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8110470 con 0x7f1db8105530 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.393+0000 7f1dbfc59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db8110770 con 0x7f1db8199500 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39318/0 (socket says 172.21.15.73:39318) 2024-03-09T00:17:24.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 -- 172.21.15.73:0/3799209233 learned_addr learned my addr 172.21.15.73:0/3799209233 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd1f4640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8105530 0x7f1db8198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbe1f6640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 0x7f1db81a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 -- 172.21.15.73:0/3799209233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 msgr2=0x7f1db81a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 0x7f1db81a3ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 -- 172.21.15.73:0/3799209233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8105530 msgr2=0x7f1db8198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8105530 0x7f1db8198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 -- 172.21.15.73:0/3799209233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1db81a42b0 con 0x7f1db8100460 2024-03-09T00:17:24.397 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbe1f6640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 0x7f1db81a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbd9f5640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1db400eab0 tx=0x7f1db400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db400ce40 con 0x7f1db8100460 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1db4011950 con 0x7f1db8100460 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db81a45a0 con 0x7f1db8100460 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1db4010ae0 con 0x7f1db8100460 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.394+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db81a4b10 con 0x7f1db8100460 2024-03-09T00:17:24.398 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.395+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db80680e0 con 0x7f1db8100460 2024-03-09T00:17:24.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.396+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1db4011b00 con 0x7f1db8100460 2024-03-09T00:17:24.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.397+0000 7f1daeffd640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 0x7f1da40a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.399 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.397+0000 7f1dbd1f4640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 0x7f1da40a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.397+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f1db40cf8c0 con 0x7f1db8100460 2024-03-09T00:17:24.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.397+0000 7f1dbd1f4640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 0x7f1da40a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1da802c240 tx=0x7f1da802df90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:24.400 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.401+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1db4016030 con 0x7f1db8100460 2024-03-09T00:17:24.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:24 smithi102 ceph-mon[36106]: pgmap v329: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:17:24.574 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.575+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1db8108790 con 0x7f1db8100460 2024-03-09T00:17:24.575 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.576+0000 7f1daeffd640 1 -- 172.21.15.73:0/3799209233 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+25346 (secure 0 0 0) 0x7f1db409c230 con 0x7f1db8100460 2024-03-09T00:17:24.576 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:17:24.576 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":104,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:17:14.975361+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/2159650606":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/102170643":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3270366724":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/2748563172":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 msgr2=0x7f1da40a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 0x7f1da40a4eb0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f1da802c240 tx=0x7f1da802df90 comp rx=0 tx=0).stop 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 msgr2=0x7f1db8198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 crypto rx=0x7f1db400eab0 tx=0x7f1db400ef70 comp rx=0 tx=0).stop 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 shutdown_connections 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1db8199500 0x7f1db81a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.578 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1db8105530 0x7f1db8198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1da40a29f0 0x7f1da40a4eb0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 --2- 172.21.15.73:0/3799209233 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1db8100460 0x7f1db8198a30 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.578+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 >> 172.21.15.73:0/3799209233 conn(0x7f1db80772f0 msgr2=0x7f1db80fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:24.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.579+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 shutdown_connections 2024-03-09T00:17:24.579 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.579+0000 7f1dbfc59640 1 -- 172.21.15.73:0/3799209233 wait complete. 2024-03-09T00:17:24.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:24 smithi179 ceph-mon[36637]: pgmap v329: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:17:24.587 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:24 smithi073 ceph-mon[32309]: pgmap v329: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 102 B/s wr, 2 op/s 2024-03-09T00:17:24.589 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:17:24.590 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:17:24.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 -- 172.21.15.73:0/3921997565 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3828101eb0 msgr2=0x7f3828105ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.697 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 --2- 172.21.15.73:0/3921997565 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3828101eb0 0x7f3828105ef0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 crypto rx=0x7f3810009900 tx=0x7f381002bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 -- 172.21.15.73:0/3921997565 shutdown_connections 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 --2- 172.21.15.73:0/3921997565 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f38281065d0 0x7f382810e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 --2- 172.21.15.73:0/3921997565 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f3828101eb0 0x7f3828105ef0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 --2- 172.21.15.73:0/3921997565 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828101500 0x7f38281018e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 -- 172.21.15.73:0/3921997565 >> 172.21.15.73:0/3921997565 conn(0x7f38280fcd20 msgr2=0x7f38280ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 -- 172.21.15.73:0/3921997565 shutdown_connections 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.696+0000 7f382d990640 1 -- 172.21.15.73:0/3921997565 wait complete. 2024-03-09T00:17:24.698 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 Processor -- start 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 -- start start 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3828101500 0x7f382819d8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828197aa0 0x7f3828197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38281116c0 con 0x7f38281065d0 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828111540 con 0x7f3828197aa0 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f382d990640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3828111840 con 0x7f3828101500 2024-03-09T00:17:24.699 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f381e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f381e7fc640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39326/0 (socket says 172.21.15.73:39326) 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f381e7fc640 1 -- 172.21.15.73:0/3661713446 learned_addr learned my addr 172.21.15.73:0/3661713446 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.697+0000 7f3826ffd640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3828101500 0x7f382819d8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f38277fe640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828197aa0 0x7f3828197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 -- 172.21.15.73:0/3661713446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3828101500 msgr2=0x7f382819d8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3828101500 0x7f382819d8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 -- 172.21.15.73:0/3661713446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828197aa0 msgr2=0x7f3828197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828197aa0 0x7f3828197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 -- 172.21.15.73:0/3661713446 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38281985c0 con 0x7f38281065d0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f381e7fc640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f381002cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810041070 con 0x7f38281065d0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3828198850 con 0x7f38281065d0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3810004450 con 0x7f38281065d0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3810034440 con 0x7f38281065d0 2024-03-09T00:17:24.701 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.698+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38281a4aa0 con 0x7f38281065d0 2024-03-09T00:17:24.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.699+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38281029d0 con 0x7f38281065d0 2024-03-09T00:17:24.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.700+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3810044070 con 0x7f38281065d0 2024-03-09T00:17:24.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.700+0000 7f382c98e640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 0x7f38140a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.702 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.701+0000 7f3826ffd640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 0x7f38140a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.701+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f38100f0f10 con 0x7f38281065d0 2024-03-09T00:17:24.703 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.701+0000 7f3826ffd640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 0x7f38140a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3818005fd0 tx=0x7f3818005e40 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:24.704 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.705+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f38100bd990 con 0x7f38281065d0 2024-03-09T00:17:24.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.878+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3828101eb0 con 0x7f38281065d0 2024-03-09T00:17:24.877 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.878+0000 7f382c98e640 1 -- 172.21.15.73:0/3661713446 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f38100bd290 con 0x7f38281065d0 2024-03-09T00:17:24.878 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.880+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 msgr2=0x7f38140a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.880+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 0x7f38140a4e30 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f3818005fd0 tx=0x7f3818005e40 comp rx=0 tx=0).stop 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 msgr2=0x7f382819de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 crypto rx=0x7f3810000c00 tx=0x7f381002cc70 comp rx=0 tx=0).stop 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 shutdown_connections 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f3828197aa0 0x7f3828197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.880 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f38140a2970 0x7f38140a4e30 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f38281065d0 0x7f382819de30 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 --2- 172.21.15.73:0/3661713446 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3828101500 0x7f382819d8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 >> 172.21.15.73:0/3661713446 conn(0x7f38280fcd20 msgr2=0x7f38280fe090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:24.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 shutdown_connections 2024-03-09T00:17:24.881 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.881+0000 7f382d990640 1 -- 172.21.15.73:0/3661713446 wait complete. 2024-03-09T00:17:24.890 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:17:24.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 -- 172.21.15.73:0/4251838301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad3406a540 msgr2=0x7fad34105ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:24.997 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 --2- 172.21.15.73:0/4251838301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad3406a540 0x7fad34105ed0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 crypto rx=0x7fad24009900 tx=0x7fad2402bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 -- 172.21.15.73:0/4251838301 shutdown_connections 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 --2- 172.21.15.73:0/4251838301 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad341065b0 0x7fad3410e190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 --2- 172.21.15.73:0/4251838301 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad3406a540 0x7fad34105ed0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 --2- 172.21.15.73:0/4251838301 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad34069b90 0x7fad34069f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 -- 172.21.15.73:0/4251838301 >> 172.21.15.73:0/4251838301 conn(0x7fad34076fb0 msgr2=0x7fad340773c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 -- 172.21.15.73:0/4251838301 shutdown_connections 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.996+0000 7fad3c162640 1 -- 172.21.15.73:0/4251838301 wait complete. 2024-03-09T00:17:24.998 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 Processor -- start 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 -- start start 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad34069b90 0x7fad3419d720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 0x7fad34197d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad341116b0 con 0x7fad341065b0 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad34111530 con 0x7fad341978d0 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3c162640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad34111830 con 0x7fad34069b90 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:24.999 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39354/0 (socket says 172.21.15.73:39354) 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 -- 172.21.15.73:0/555344668 learned_addr learned my addr 172.21.15.73:0/555344668 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3a6ff640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 0x7fad34197d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 -- 172.21.15.73:0/555344668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad34069b90 msgr2=0x7fad3419d720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad34069b90 0x7fad3419d720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 -- 172.21.15.73:0/555344668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 msgr2=0x7fad34197d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 0x7fad34197d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 -- 172.21.15.73:0/555344668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad341983f0 con 0x7fad341065b0 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad3a6ff640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 0x7fad34197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:25.000 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad396fd640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2402cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.997+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24041070 con 0x7fad341065b0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.998+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad34198680 con 0x7fad341065b0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.998+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fad24004450 con 0x7fad341065b0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.998+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fad24034440 con 0x7fad341065b0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.998+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad341a4870 con 0x7fad341065b0 2024-03-09T00:17:25.001 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:24.998+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad341029b0 con 0x7fad341065b0 2024-03-09T00:17:25.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.000+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fad24044070 con 0x7fad341065b0 2024-03-09T00:17:25.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.000+0000 7fad2affd640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 0x7fad200a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.002 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.000+0000 7fad39efe640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 0x7fad200a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.000+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fad240f11b0 con 0x7fad341065b0 2024-03-09T00:17:25.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.001+0000 7fad39efe640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 0x7fad200a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fad34198a10 tx=0x7fad30005c70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.005 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.004+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fad240bdc30 con 0x7fad341065b0 2024-03-09T00:17:25.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.179+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fad34198e00 con 0x7fad341065b0 2024-03-09T00:17:25.178 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.179+0000 7fad2affd640 1 -- 172.21.15.73:0/555344668 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7fad240bd530 con 0x7fad341065b0 2024-03-09T00:17:25.178 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:17:25.180 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 msgr2=0x7fad200a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 0x7fad200a4e30 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fad34198a10 tx=0x7fad30005c70 comp rx=0 tx=0).stop 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 msgr2=0x7fad3419dc60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 crypto rx=0x7fad24000c00 tx=0x7fad2402cc70 comp rx=0 tx=0).stop 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 shutdown_connections 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fad341978d0 0x7fad34197d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fad200a2970 0x7fad200a4e30 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fad341065b0 0x7fad3419dc60 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.181 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 --2- 172.21.15.73:0/555344668 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fad34069b90 0x7fad3419d720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.182 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 >> 172.21.15.73:0/555344668 conn(0x7fad34076fb0 msgr2=0x7fad340fe4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.182 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 shutdown_connections 2024-03-09T00:17:25.182 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.181+0000 7fad3c162640 1 -- 172.21.15.73:0/555344668 wait complete. 2024-03-09T00:17:25.191 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:17:25.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- 172.21.15.73:0/2873999727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601003d0 msgr2=0x7fa8601007b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- 172.21.15.73:0/2873999727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601003d0 0x7fa8601007b0 secure :-1 s=READY pgs=1040 cs=0 l=1 rev1=1 crypto rx=0x7fa854009900 tx=0x7fa85402bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:25.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- 172.21.15.73:0/2873999727 shutdown_connections 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- 172.21.15.73:0/2873999727 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa8601054a0 0x7fa86010d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- 172.21.15.73:0/2873999727 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa860100d80 0x7fa860104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- 172.21.15.73:0/2873999727 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601003d0 0x7fa8601007b0 unknown :-1 s=CLOSED pgs=1040 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- 172.21.15.73:0/2873999727 >> 172.21.15.73:0/2873999727 conn(0x7fa8600772f0 msgr2=0x7fa860077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- 172.21.15.73:0/2873999727 shutdown_connections 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- 172.21.15.73:0/2873999727 wait complete. 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 Processor -- start 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- start start 2024-03-09T00:17:25.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 0x7fa860198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa860199500 0x7fa8601a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8601105b0 con 0x7fa8601054a0 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa860110430 con 0x7fa8601003d0 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa867174640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa860110730 con 0x7fa860199500 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.297+0000 7fa85ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39376/0 (socket says 172.21.15.73:39376) 2024-03-09T00:17:25.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 -- 172.21.15.73:0/1742863493 learned_addr learned my addr 172.21.15.73:0/1742863493 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa864f10640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 0x7fa860198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 -- 172.21.15.73:0/1742863493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa860199500 msgr2=0x7fa8601a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa865711640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa860199500 0x7fa8601a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa860199500 0x7fa8601a3ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 -- 172.21.15.73:0/1742863493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 msgr2=0x7fa860198a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 0x7fa860198a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 -- 172.21.15.73:0/1742863493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8601a42b0 con 0x7fa8601054a0 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa864f10640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 0x7fa860198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:17:25.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85ffff640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c970 tx=0x7fa84c00ce30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c007bb0 con 0x7fa8601054a0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8601a45a0 con 0x7fa8601054a0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa84c007d50 con 0x7fa8601054a0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa84c00f450 con 0x7fa8601054a0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.298+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8601a4b40 con 0x7fa8601054a0 2024-03-09T00:17:25.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.299+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8600680e0 con 0x7fa8601054a0 2024-03-09T00:17:25.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.300+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa84c004f90 con 0x7fa8601054a0 2024-03-09T00:17:25.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.301+0000 7fa85dffb640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.303 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.301+0000 7fa864f10640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.301+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fa84c0cf070 con 0x7fa8601054a0 2024-03-09T00:17:25.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.301+0000 7fa864f10640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 0x7fa8500a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa854036040 tx=0x7fa854000bc0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.304 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.306+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa84c014030 con 0x7fa8601054a0 2024-03-09T00:17:25.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.480+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa8601086d0 con 0x7fa8601054a0 2024-03-09T00:17:25.479 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.480+0000 7fa85dffb640 1 -- 172.21.15.73:0/1742863493 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa84c09b9e0 con 0x7fa8601054a0 2024-03-09T00:17:25.479 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:17:25.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 msgr2=0x7fa8500a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.481 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 0x7fa8500a4e30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fa854036040 tx=0x7fa854000bc0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 msgr2=0x7fa860198f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 crypto rx=0x7fa84c00c970 tx=0x7fa84c00ce30 comp rx=0 tx=0).stop 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 shutdown_connections 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa860199500 0x7fa8601a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa8500a2970 0x7fa8500a4e30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa8601054a0 0x7fa860198f70 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.482 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 --2- 172.21.15.73:0/1742863493 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa8601003d0 0x7fa860198a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 >> 172.21.15.73:0/1742863493 conn(0x7fa8600772f0 msgr2=0x7fa8600fe370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 shutdown_connections 2024-03-09T00:17:25.483 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.482+0000 7fa867174640 1 -- 172.21.15.73:0/1742863493 wait complete. 2024-03-09T00:17:25.491 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:25 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3799209233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:17:25.491 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:25 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3661713446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:17:25.492 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:17:25.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:25 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3799209233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:17:25.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:25 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3661713446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:17:25.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:25 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3799209233' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:17:25.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:25 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3661713446' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:17:25.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.597+0000 7feb794e9640 1 -- 172.21.15.73:0/981648272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74101500 msgr2=0x7feb741018e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.597+0000 7feb794e9640 1 --2- 172.21.15.73:0/981648272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74101500 0x7feb741018e0 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 crypto rx=0x7feb64009ca0 tx=0x7feb6402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 -- 172.21.15.73:0/981648272 shutdown_connections 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- 172.21.15.73:0/981648272 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb741065d0 0x7feb7410e1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- 172.21.15.73:0/981648272 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb74101eb0 0x7feb74105ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- 172.21.15.73:0/981648272 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74101500 0x7feb741018e0 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 -- 172.21.15.73:0/981648272 >> 172.21.15.73:0/981648272 conn(0x7feb740fcd20 msgr2=0x7feb740ff140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 -- 172.21.15.73:0/981648272 shutdown_connections 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 -- 172.21.15.73:0/981648272 wait complete. 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 Processor -- start 2024-03-09T00:17:25.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 -- start start 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 0x7feb7419d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb741065d0 0x7feb7419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.598+0000 7feb794e9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb794e9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb741116e0 con 0x7feb74197a60 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb794e9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74111560 con 0x7feb74101500 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb794e9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb74111860 con 0x7feb741065d0 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.601 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39396/0 (socket says 172.21.15.73:39396) 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 -- 172.21.15.73:0/2771766586 learned_addr learned my addr 172.21.15.73:0/2771766586 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb72ffd640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 0x7feb7419d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 -- 172.21.15.73:0/2771766586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb741065d0 msgr2=0x7feb7419ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb727fc640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb741065d0 0x7feb7419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb741065d0 0x7feb7419ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 -- 172.21.15.73:0/2771766586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 msgr2=0x7feb7419d8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 0x7feb7419d8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.602 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 -- 172.21.15.73:0/2771766586 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb74198580 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb72ffd640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 0x7feb7419d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb737fe640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00d950 tx=0x7feb6c00de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c017030 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb74198870 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feb6c004020 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feb6c003c00 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.599+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb741a4ac0 con 0x7feb74197a60 2024-03-09T00:17:25.603 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.600+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb74068530 con 0x7feb74197a60 2024-03-09T00:17:25.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.601+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7feb6c01a070 con 0x7feb74197a60 2024-03-09T00:17:25.604 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.602+0000 7feb5bfff640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 0x7feb5c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.602+0000 7feb72ffd640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 0x7feb5c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.602+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7feb6c0cec40 con 0x7feb74197a60 2024-03-09T00:17:25.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.603+0000 7feb72ffd640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 0x7feb5c0a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb6402c700 tx=0x7feb64005c50 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.605 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.607+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7feb6c021070 con 0x7feb74197a60 2024-03-09T00:17:25.778 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.779+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7feb74101eb0 con 0x7feb74197a60 2024-03-09T00:17:25.779 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.780+0000 7feb5bfff640 1 -- 172.21.15.73:0/2771766586 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) v1 ==== 103+0+11 (secure 0 0 0) 0x7feb6c09b5b0 con 0x7feb74197a60 2024-03-09T00:17:25.779 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.781+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 msgr2=0x7feb5c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.781+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 0x7feb5c0a4e30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7feb6402c700 tx=0x7feb64005c50 comp rx=0 tx=0).stop 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.781+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 msgr2=0x7feb74197f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 crypto rx=0x7feb6c00d950 tx=0x7feb6c00de10 comp rx=0 tx=0).stop 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 shutdown_connections 2024-03-09T00:17:25.781 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7feb5c0a2970 0x7feb5c0a4e30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7feb74197a60 0x7feb74197f10 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7feb741065d0 0x7feb7419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 --2- 172.21.15.73:0/2771766586 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7feb74101500 0x7feb7419d8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 >> 172.21.15.73:0/2771766586 conn(0x7feb740fcd20 msgr2=0x7feb740fe070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 shutdown_connections 2024-03-09T00:17:25.782 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.782+0000 7feb794e9640 1 -- 172.21.15.73:0/2771766586 wait complete. 2024-03-09T00:17:25.792 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-09T00:17:25.892 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.893+0000 7f2453e32640 1 -- 172.21.15.73:0/3983477465 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 msgr2=0x7f244c100840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.893 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.893+0000 7f2453e32640 1 --2- 172.21.15.73:0/3983477465 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c100840 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 crypto rx=0x7f2448009900 tx=0x7f244802bc30 comp rx=0 tx=0).stop 2024-03-09T00:17:25.895 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- 172.21.15.73:0/3983477465 shutdown_connections 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- 172.21.15.73:0/3983477465 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 0x7f244c10d100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- 172.21.15.73:0/3983477465 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f244c100e10 0x7f244c104e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- 172.21.15.73:0/3983477465 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c100840 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- 172.21.15.73:0/3983477465 >> 172.21.15.73:0/3983477465 conn(0x7f244c0772f0 msgr2=0x7f244c077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- 172.21.15.73:0/3983477465 shutdown_connections 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- 172.21.15.73:0/3983477465 wait complete. 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 Processor -- start 2024-03-09T00:17:25.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- start start 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 0x7f244c198f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f244c199550 0x7f244c1a3ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c1105f0 con 0x7f244c100460 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c110470 con 0x7f244c199550 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2453e32640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f244c110770 con 0x7f244c105530 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2451bce640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2451bce640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:39406/0 (socket says 172.21.15.73:39406) 2024-03-09T00:17:25.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.894+0000 7f2451bce640 1 -- 172.21.15.73:0/709039941 learned_addr learned my addr 172.21.15.73:0/709039941 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f24513cd640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 0x7f244c198f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 -- 172.21.15.73:0/709039941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 msgr2=0x7f244c198f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 0x7f244c198f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 -- 172.21.15.73:0/709039941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f244c199550 msgr2=0x7f244c1a3ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f244c199550 0x7f244c1a3ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 -- 172.21.15.73:0/709039941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f244c1a42d0 con 0x7f244c100460 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2451bce640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2448006010 tx=0x7f244802cd70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448041070 con 0x7f244c100460 2024-03-09T00:17:25.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f24480054e0 con 0x7f244c100460 2024-03-09T00:17:25.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f244c1a4560 con 0x7f244c100460 2024-03-09T00:17:25.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2448034440 con 0x7f244c100460 2024-03-09T00:17:25.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.895+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f244c1a4a50 con 0x7f244c100460 2024-03-09T00:17:25.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.896+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f244c0680e0 con 0x7f244c100460 2024-03-09T00:17:25.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.897+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f2448044070 con 0x7f244c100460 2024-03-09T00:17:25.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.897+0000 7f2442ffd640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 0x7f24380a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:17:25.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.898+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f24480f0e40 con 0x7f244c100460 2024-03-09T00:17:25.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.898+0000 7f24513cd640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 0x7f24380a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:17:25.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.898+0000 7f24513cd640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 0x7f24380a4e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f244c069000 tx=0x7f243c00a4c0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:17:25.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:25.902+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f24480bd8c0 con 0x7f244c100460 2024-03-09T00:17:26.126 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.127+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f244c108760 con 0x7f244c100460 2024-03-09T00:17:26.133 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.134+0000 7f2442ffd640 1 -- 172.21.15.73:0/709039941 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v26) v1 ==== 81+0+740361 (secure 0 0 0) 0x7f24480bd1c0 con 0x7f244c100460 2024-03-09T00:17:26.134 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:17:26.134 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:26.134 INFO:teuthology.orchestra.run.smithi073.stdout: "epoch": 26, 2024-03-09T00:17:26.134 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "active_gid": 15564, 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "active_name": "x", 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "active_addrs": { 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:6800", 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 2716694463 2024-03-09T00:17:26.135 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v1", 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:6801", 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 2716694463 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.136 INFO:teuthology.orchestra.run.smithi073.stdout: "active_addr": "172.21.15.73:6801/2716694463", 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "active_change": "2024-03-09T00:06:50.943197+0000", 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "active_mgr_features": 4540701547738038271, 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "available": true, 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "standbys": [ 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "gid": 25055, 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "y", 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr_features": 4540701547738038271, 2024-03-09T00:17:26.137 INFO:teuthology.orchestra.run.smithi073.stdout: "available_modules": [ 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "alerts", 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.138 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How frequently to reexamine health status", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.139 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.140 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.141 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.142 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.143 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.144 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.145 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.146 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.147 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_destination": { 2024-03-09T00:17:26.148 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_destination", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.149 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email address to send alerts to", 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_from_name": { 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_from_name", 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.150 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "Ceph", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email From: name", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.151 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_host": { 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_host", 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.152 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP server", 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_password": { 2024-03-09T00:17:26.153 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_password", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.154 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Password to authenticate with", 2024-03-09T00:17:26.155 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.155 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_port": { 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_port", 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.156 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "465", 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP port", 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.157 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_sender": { 2024-03-09T00:17:26.158 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_sender", 2024-03-09T00:17:26.158 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.158 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.158 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP envelope sender", 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.159 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_ssl": { 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_ssl", 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.160 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_user": { 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_user", 2024-03-09T00:17:26.161 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.162 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "User to authenticate as", 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.163 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "balancer", 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "active": { 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "active", 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.164 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically balance PGs across cluster", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.165 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_time": { 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_time", 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0000", 2024-03-09T00:17:26.166 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "beginning time of day to automatically balance", 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.167 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_weekday": { 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_weekday", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.168 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_max_iterations": { 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_max_iterations", 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.169 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "25", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "250", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.170 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_metrics": { 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_metrics", 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.171 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "pgs,objects,bytes", 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.172 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_step": { 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_step", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.5", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0.001", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "0.999", 2024-03-09T00:17:26.173 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "aggressiveness of optimization", 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "end_time": { 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_time", 2024-03-09T00:17:26.174 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2359", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ending time of day to automatically balance", 2024-03-09T00:17:26.175 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "end_weekday": { 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_weekday", 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.176 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.177 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.178 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.179 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.180 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.181 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.190 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.191 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.192 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.193 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.194 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "min_score": { 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_score", 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.195 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.196 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "upmap", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.197 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "crush-compat", 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "none", 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "read", 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap", 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap-read" 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Balancer mode", 2024-03-09T00:17:26.198 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_ids": { 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_ids", 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.199 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.200 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.201 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_deviation": { 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_deviation", 2024-03-09T00:17:26.202 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-09T00:17:26.203 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_optimizations": { 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_optimizations", 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.204 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.205 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm", 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.206 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_down_multiplier": { 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_down_multiplier", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.207 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_refresh_rate": { 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_refresh_rate", 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.208 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "20", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.209 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_starting_port": { 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_starting_port", 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4721", 2024-03-09T00:17:26.210 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.211 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_ptrace": { 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_ptrace", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.212 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_interval": { 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_interval", 2024-03-09T00:17:26.213 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.214 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_memory_target_ratio": { 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_memory_target_ratio", 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.215 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.7", 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.216 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm_log_destination": { 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm_log_destination", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.217 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "file", 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "file,syslog", 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "syslog" 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.218 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "cgroups_split": { 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cgroups_split", 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.219 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "config_checks_enabled": { 2024-03-09T00:17:26.220 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_checks_enabled", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.221 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "config_dashboard": { 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_dashboard", 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.222 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.223 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_alertmanager": { 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_alertmanager", 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.224 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_base": { 2024-03-09T00:17:26.225 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_base", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.226 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Container image name, without the tag", 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_elasticsearch": { 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_elasticsearch", 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.227 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "elasticsearch container image", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.228 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_grafana": { 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_grafana", 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-09T00:17:26.229 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.230 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_haproxy": { 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_haproxy", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.231 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "HAproxy container image", 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_agent": { 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_agent", 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.232 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger agent container image", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.233 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_collector": { 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_collector", 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.234 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger collector container image", 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.235 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_query": { 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_query", 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-09T00:17:26.236 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger query container image", 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_keepalived": { 2024-03-09T00:17:26.237 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_keepalived", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.238 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Keepalived container image", 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_loki": { 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_loki", 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.239 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Loki container image", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.240 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_node_exporter": { 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_node_exporter", 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-09T00:17:26.241 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.242 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_nvmeof": { 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_nvmeof", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.243 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Nvme-of container image", 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_prometheus": { 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_prometheus", 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.244 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.245 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_promtail": { 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_promtail", 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-09T00:17:26.246 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Promtail container image", 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.247 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_snmp_gateway": { 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_snmp_gateway", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.248 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SNMP Gateway container image", 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "container_init": { 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_init", 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.249 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Run podman/docker with `--init`", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.250 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "daemon_cache_timeout": { 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "daemon_cache_timeout", 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.251 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.252 INFO:teuthology.orchestra.run.smithi073.stdout: "default_cephadm_command_timeout": { 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_cephadm_command_timeout", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "900", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.253 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "default_registry": { 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_registry", 2024-03-09T00:17:26.254 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.255 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "device_cache_timeout": { 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_cache_timeout", 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.256 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1800", 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache device inventory", 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.257 INFO:teuthology.orchestra.run.smithi073.stdout: "device_enhanced_scan": { 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_enhanced_scan", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.258 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "facts_cache_timeout": { 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "facts_cache_timeout", 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.259 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache host facts data", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.260 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "host_check_interval": { 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "host_check_interval", 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.261 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to perform a host check", 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.262 INFO:teuthology.orchestra.run.smithi073.stdout: "hw_monitoring": { 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hw_monitoring", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.263 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "inventory_list_all": { 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "inventory_list_all", 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.264 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.265 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.266 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.267 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "log_refresh_metadata": { 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_refresh_metadata", 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.268 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.269 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.270 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.271 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.272 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.273 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.274 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.275 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.276 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.277 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "*", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.278 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "max_count_per_host": { 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_count_per_host", 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.279 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of daemons per service per host", 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.280 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "max_osd_draining_count": { 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_osd_draining_count", 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.281 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "migration_current": { 2024-03-09T00:17:26.282 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "migration_current", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "internal - do not modify", 2024-03-09T00:17:26.283 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.284 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "root", 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm-package", 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "root" 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "mode for remote execution of cephadm", 2024-03-09T00:17:26.285 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "oob_default_addr": { 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "oob_default_addr", 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.286 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "169.254.1.1", 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.287 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "prometheus_alerts_path": { 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus_alerts_path", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.288 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_insecure": { 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_insecure", 2024-03-09T00:17:26.289 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.290 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_password": { 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_password", 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.291 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.292 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_url": { 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_url", 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.293 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_username": { 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_username", 2024-03-09T00:17:26.294 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-09T00:17:26.295 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "secure_monitoring_stack": { 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "secure_monitoring_stack", 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.296 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.297 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "service_discovery_port": { 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "service_discovery_port", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8765", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.298 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "cephadm service discovery port", 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "ssh_config_file": { 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssh_config_file", 2024-03-09T00:17:26.299 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.300 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "use_agent": { 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_agent", 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.301 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.302 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "use_repo_digest": { 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_repo_digest", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.303 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_failed_host_check": { 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_failed_host_check", 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.304 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if the host check fails", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.305 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.306 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.306 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.306 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_daemons": { 2024-03-09T00:17:26.306 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_daemons", 2024-03-09T00:17:26.306 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.307 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_hosts": { 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_hosts", 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.308 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.309 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crash", 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.310 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.311 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.312 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.313 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.314 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.315 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.316 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.317 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.318 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "retain_interval": { 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retain_interval", 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.319 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "31536000", 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.320 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_recent_interval": { 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_recent_interval", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1209600", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.321 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.322 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dashboard", 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.323 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.324 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_HOST": { 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.325 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-09T00:17:26.326 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.327 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_ENABLED": { 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_ENABLED", 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.328 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.329 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.330 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "ENABLE_BROWSABLE_API": { 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-09T00:17:26.331 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.332 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.333 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.334 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.335 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-09T00:17:26.336 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.337 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.338 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.339 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.340 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-09T00:17:26.341 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.342 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.343 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.344 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.345 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_PASSWORD": { 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-09T00:17:26.346 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.347 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.348 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.349 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_URL": { 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_URL", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.350 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_USERNAME": { 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.351 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.352 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.353 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.354 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.355 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.356 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.357 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.358 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.359 INFO:teuthology.orchestra.run.smithi073.stdout: "MULTICLUSTER_CONFIG": { 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "{}", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.360 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_HOST": { 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.361 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.362 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.363 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.364 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.365 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.366 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.367 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.368 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-09T00:17:26.369 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.370 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-09T00:17:26.371 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.372 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.372 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.372 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.372 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.374 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.375 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.376 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.377 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_ENABLED": { 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.378 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.379 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.380 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-09T00:17:26.381 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.382 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.383 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8", 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.384 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "45", 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.385 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ACCESS_KEY": { 2024-03-09T00:17:26.386 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.387 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.388 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.389 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SECRET_KEY": { 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.390 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SSL_VERIFY": { 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-09T00:17:26.391 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.392 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "UNSAFE_TLS_v1_2": { 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.393 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.394 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.395 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-09T00:17:26.396 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.397 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.398 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.399 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "cross_origin_url": { 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cross_origin_url", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.400 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "crt_file": { 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crt_file", 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.401 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.402 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "debug": { 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "debug", 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.403 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable/disable debug options", 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.404 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "jwt_token_ttl": { 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "jwt_token_ttl", 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "28800", 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.405 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:26.406 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.407 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.408 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.409 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.410 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.411 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.412 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.413 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.414 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.415 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "motd": { 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "motd", 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.416 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "The message of the day", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.417 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect_resolve_ip_addr": { 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "redirect_resolve_ip_addr", 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.418 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.419 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.420 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.421 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8080", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.422 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.423 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl_server_port": { 2024-03-09T00:17:26.424 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl_server_port", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8443", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.425 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.426 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "redirect", 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect" 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.427 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.428 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.429 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "url_prefix": { 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url_prefix", 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.430 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.431 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "devicehealth", 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_monitoring": { 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_monitoring", 2024-03-09T00:17:26.432 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "monitor device health metrics", 2024-03-09T00:17:26.433 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.434 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.435 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.436 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.437 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.438 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.439 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.440 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.441 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.442 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "mark_out_threshold": { 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mark_out_threshold", 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2419200", 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.443 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_name": { 2024-03-09T00:17:26.444 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_name", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "device_health_metrics", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-09T00:17:26.445 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "retention_period": { 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retention_period", 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.446 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15552000", 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain device health metrics", 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.447 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_frequency": { 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_frequency", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.448 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "self_heal": { 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "self_heal", 2024-03-09T00:17:26.449 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.450 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.451 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and check device health", 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.452 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_threshold": { 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_threshold", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7257600", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.453 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "diskprediction_local", 2024-03-09T00:17:26.454 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.455 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.456 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.457 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.458 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.459 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.460 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.461 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.462 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "predict_interval": { 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predict_interval", 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:26.463 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "predictor_model": { 2024-03-09T00:17:26.464 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predictor_model", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "prophetstor", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.465 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.466 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.467 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "influx", 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": false, 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "influxdb python module not found", 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "batch_size": { 2024-03-09T00:17:26.468 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "batch_size", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5000", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.469 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "database": { 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "database", 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.470 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "ceph", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.471 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "hostname": { 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hostname", 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.472 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server hostname", 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.473 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "30", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "5", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.474 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.475 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.476 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.477 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.478 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.479 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.480 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.481 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.482 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "password": { 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "password", 2024-03-09T00:17:26.483 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "password of InfluxDB server user", 2024-03-09T00:17:26.484 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "port": { 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "port", 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.485 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8086", 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server port", 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.486 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "false", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.487 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "threads": { 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threads", 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.488 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "32", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.489 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "username": { 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "username", 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.490 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "username of InfluxDB server user", 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "verify_ssl": { 2024-03-09T00:17:26.491 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "verify_ssl", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "true", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.492 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "insights", 2024-03-09T00:17:26.493 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.494 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.495 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.496 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.497 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.498 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.499 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.500 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.501 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "iostat", 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.502 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.503 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.504 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.505 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.506 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.507 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.508 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.509 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.510 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "k8sevents", 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_event_retention_days": { 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ceph_event_retention_days", 2024-03-09T00:17:26.511 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-09T00:17:26.512 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "config_check_secs": { 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_check_secs", 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.513 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "10", 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.514 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.515 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.516 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.517 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.518 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.519 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.520 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.521 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.522 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "localpool", 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.523 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "failure_domain": { 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "failure_domain", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "host", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.524 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "failure domain for any created local pool", 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.525 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.526 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.527 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.528 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.529 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.530 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.531 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.532 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.533 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "min_size": { 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_size", 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.534 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default min_size for any created local pool", 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "num_rep": { 2024-03-09T00:17:26.535 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "num_rep", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default replica count for any created local pool", 2024-03-09T00:17:26.536 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_num": { 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_num", 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.537 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "128", 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default pg_num for any created local pool", 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.538 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "prefix": { 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prefix", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.539 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name prefix for any created local pool", 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "subtree": { 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "subtree", 2024-03-09T00:17:26.540 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "rack", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-09T00:17:26.541 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mds_autoscaler", 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.542 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.543 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.544 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.545 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.546 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.547 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.548 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.549 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.550 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirroring", 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.551 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.552 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.553 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.554 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.555 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.556 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.557 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.558 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.559 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.560 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "nfs", 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.561 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.562 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.563 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.564 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.565 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.566 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.567 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.568 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:26.569 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "fail_fs": { 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "fail_fs", 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.570 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.571 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.572 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.573 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.574 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.575 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.576 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.577 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.578 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.579 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator": { 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.580 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm", 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "rook", 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "test_orchestrator" 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Orchestrator backend", 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.581 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_perf_query", 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.582 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.583 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.584 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.585 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.586 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.587 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.588 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.589 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.590 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_support", 2024-03-09T00:17:26.591 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.592 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.593 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.594 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.595 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.596 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.597 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.598 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.599 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_autoscaler", 2024-03-09T00:17:26.600 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.601 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.602 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.603 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.604 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.605 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.606 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.607 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.608 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.609 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.610 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "threshold": { 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threshold", 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:26.611 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1.0", 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "scaling threshold", 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.612 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "progress", 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_pg_recovery_event": { 2024-03-09T00:17:26.613 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_pg_recovery_event", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-09T00:17:26.614 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.615 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.616 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.617 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.618 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.619 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.620 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.621 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.622 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.623 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.625 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.625 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.625 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.625 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.625 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.626 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "max_completed_events": { 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_completed_events", 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "50", 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.627 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "number of past completed events to remember", 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.628 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long the module is going to sleep", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.629 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus", 2024-03-09T00:17:26.630 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "cache": { 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cache", 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.631 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.632 INFO:teuthology.orchestra.run.smithi073.stdout: "exclude_perf_counters": { 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "exclude_perf_counters", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.633 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.634 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.635 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.636 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.637 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.638 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.639 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.640 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.641 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.642 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools": { 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.643 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.644 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "300", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.645 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_interval": { 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_interval", 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15.0", 2024-03-09T00:17:26.646 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.647 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.648 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.649 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "9283", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.650 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "stale_cache_strategy": { 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stale_cache_strategy", 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "log", 2024-03-09T00:17:26.651 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.652 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "default", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.653 INFO:teuthology.orchestra.run.smithi073.stdout: "default", 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "error" 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.654 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.655 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.656 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_support", 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.657 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.658 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.659 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.660 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.661 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.662 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.663 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.664 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_snap_create": { 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_snap_create", 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.665 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.666 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "mirror_snapshot_schedule": { 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirror_snapshot_schedule", 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.667 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.668 INFO:teuthology.orchestra.run.smithi073.stdout: "trash_purge_schedule": { 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "trash_purge_schedule", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.669 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.670 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "restful", 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_auth": { 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_auth", 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.671 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.672 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.673 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.674 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.675 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.676 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.677 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.678 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.679 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.680 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.681 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.682 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.683 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.684 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.685 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.686 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rgw", 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.687 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.688 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.689 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.690 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.691 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.692 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.693 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.694 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.695 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rook", 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.696 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.697 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.698 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.699 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.700 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.701 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.702 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.703 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.704 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "storage_class": { 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "storage_class", 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "local", 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.705 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.706 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "selftest", 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.707 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.708 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.709 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.710 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.711 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.712 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.713 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.714 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.715 INFO:teuthology.orchestra.run.smithi073.stdout: "roption1": { 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption1", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.716 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "roption2": { 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption2", 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.717 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "xyz", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.718 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption1": { 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption1", 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.719 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption2": { 2024-03-09T00:17:26.720 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption2", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.721 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption3": { 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption3", 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.722 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.723 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption4": { 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption4", 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.724 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.725 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption5": { 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption5", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.726 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption6": { 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption6", 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.727 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.728 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption7": { 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption7", 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.729 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "42", 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.730 INFO:teuthology.orchestra.run.smithi073.stdout: "testkey": { 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testkey", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.731 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "testlkey": { 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testlkey", 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.732 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.733 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "testnewline": { 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testnewline", 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.734 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.735 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snap_schedule", 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_m_granularity": { 2024-03-09T00:17:26.736 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_m_granularity", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow minute scheduled snapshots", 2024-03-09T00:17:26.737 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "dump_on_update": { 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dump_on_update", 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.738 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "dump database to debug log on update", 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.739 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.740 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.741 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.742 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.743 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.744 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.745 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.746 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.747 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stats", 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.748 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.749 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.750 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.751 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.752 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.753 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.754 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.755 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.756 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "status", 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.757 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.758 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.759 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.760 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.761 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.762 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.763 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.764 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.765 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telegraf", 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.766 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "address": { 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "address", 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-09T00:17:26.767 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.768 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.769 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.770 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.771 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.772 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.773 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.774 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.775 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.776 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.777 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.778 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telemetry", 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_basic": { 2024-03-09T00:17:26.779 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_basic", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.780 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_crash": { 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_crash", 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.781 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.782 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_device": { 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_device", 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.783 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_ident": { 2024-03-09T00:17:26.784 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_ident", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.785 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_perf": { 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_perf", 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.786 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.787 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "contact": { 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "contact", 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.788 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "description": { 2024-03-09T00:17:26.789 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "description", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.790 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "device_url": { 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_url", 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.791 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.792 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.793 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.794 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "24", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "8", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.795 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "last_opt_revision": { 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "last_opt_revision", 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.796 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.797 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard": { 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard", 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.798 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard_description": { 2024-03-09T00:17:26.799 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard_description", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.800 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.801 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.802 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.803 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.804 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.805 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.806 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.807 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.808 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "organization": { 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "organization", 2024-03-09T00:17:26.809 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.810 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "proxy": { 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "proxy", 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.811 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.812 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "url": { 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url", 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.813 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.814 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "test_orchestrator", 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.815 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.816 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.817 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.818 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.819 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.820 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.821 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.822 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.823 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "volumes", 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.824 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.825 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.826 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.827 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.828 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.829 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.830 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.831 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_clones": { 2024-03-09T00:17:26.832 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_clones", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-09T00:17:26.833 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "periodic_async_work": { 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "periodic_async_work", 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.834 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Periodically check for async work", 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.835 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_delay": { 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_delay", 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.836 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_no_wait": { 2024-03-09T00:17:26.837 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_no_wait", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.838 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix", 2024-03-09T00:17:26.839 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "discovery_interval": { 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "discovery_interval", 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.840 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "100", 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.841 INFO:teuthology.orchestra.run.smithi073.stdout: "identifier": { 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "identifier", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.842 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.843 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.844 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.845 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.846 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.847 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.848 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.849 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.850 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.851 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.852 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_host": { 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_host", 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.853 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.854 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_port": { 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_port", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10051", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.855 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_sender": { 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_sender", 2024-03-09T00:17:26.856 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.857 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.858 INFO:teuthology.orchestra.run.smithi073.stdout: "gid": 25132, 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "z", 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "mgr_features": 4540701547738038271, 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "available_modules": [ 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "alerts", 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.859 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.860 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How frequently to reexamine health status", 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.861 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.862 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.863 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.864 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.865 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.866 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.867 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.868 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.869 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_destination": { 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_destination", 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.870 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email address to send alerts to", 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.871 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_from_name": { 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_from_name", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "Ceph", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.872 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email From: name", 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_host": { 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_host", 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.873 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP server", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.874 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_password": { 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_password", 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.875 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Password to authenticate with", 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.876 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_port": { 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_port", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "465", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.877 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP port", 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_sender": { 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_sender", 2024-03-09T00:17:26.878 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP envelope sender", 2024-03-09T00:17:26.879 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_ssl": { 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_ssl", 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.880 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.881 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_user": { 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_user", 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.882 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "User to authenticate as", 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.883 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "balancer", 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "active": { 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "active", 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.884 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically balance PGs across cluster", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.885 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_time": { 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_time", 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.886 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0000", 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "beginning time of day to automatically balance", 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.887 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_weekday": { 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_weekday", 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:26.888 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_max_iterations": { 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_max_iterations", 2024-03-09T00:17:26.889 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "25", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "250", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-09T00:17:26.890 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_metrics": { 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_metrics", 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.891 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "pgs,objects,bytes", 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.892 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_step": { 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_step", 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.5", 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0.001", 2024-03-09T00:17:26.893 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "0.999", 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "aggressiveness of optimization", 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "end_time": { 2024-03-09T00:17:26.894 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_time", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2359", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ending time of day to automatically balance", 2024-03-09T00:17:26.895 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "end_weekday": { 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_weekday", 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.896 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.897 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.898 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.899 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.900 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.901 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.902 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.903 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.904 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.905 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "min_score": { 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_score", 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.906 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.907 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "upmap", 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.908 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "crush-compat", 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "none", 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "read", 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap", 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap-read" 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.909 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Balancer mode", 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_ids": { 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_ids", 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.910 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.911 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.912 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_deviation": { 2024-03-09T00:17:26.913 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_deviation", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.914 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_optimizations": { 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_optimizations", 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:26.915 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.916 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm", 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:26.917 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_down_multiplier": { 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_down_multiplier", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.918 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_refresh_rate": { 2024-03-09T00:17:26.919 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_refresh_rate", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "20", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-09T00:17:26.920 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_starting_port": { 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_starting_port", 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.921 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4721", 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.922 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_ptrace": { 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_ptrace", 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.923 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_interval": { 2024-03-09T00:17:26.924 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_interval", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.925 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_memory_target_ratio": { 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_memory_target_ratio", 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:26.926 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.7", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.927 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm_log_destination": { 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm_log_destination", 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.928 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "file", 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "file,syslog", 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "syslog" 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.929 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "cgroups_split": { 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cgroups_split", 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.930 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.931 INFO:teuthology.orchestra.run.smithi073.stdout: "config_checks_enabled": { 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_checks_enabled", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.932 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "config_dashboard": { 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_dashboard", 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.933 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.934 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_alertmanager": { 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_alertmanager", 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-09T00:17:26.935 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.936 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_base": { 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_base", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.937 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Container image name, without the tag", 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_elasticsearch": { 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_elasticsearch", 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.938 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "elasticsearch container image", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.939 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_grafana": { 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_grafana", 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-09T00:17:26.940 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.941 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_haproxy": { 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_haproxy", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.942 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "HAproxy container image", 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_agent": { 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_agent", 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.943 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger agent container image", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.944 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_collector": { 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_collector", 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.945 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger collector container image", 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.946 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_query": { 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_query", 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.947 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger query container image", 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_keepalived": { 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_keepalived", 2024-03-09T00:17:26.948 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Keepalived container image", 2024-03-09T00:17:26.949 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_loki": { 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_loki", 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.950 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Loki container image", 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.951 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_node_exporter": { 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_node_exporter", 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.952 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_nvmeof": { 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_nvmeof", 2024-03-09T00:17:26.953 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Nvme-of container image", 2024-03-09T00:17:26.954 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_prometheus": { 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_prometheus", 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.955 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.956 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_promtail": { 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_promtail", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.957 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Promtail container image", 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_snmp_gateway": { 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_snmp_gateway", 2024-03-09T00:17:26.958 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SNMP Gateway container image", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.959 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "container_init": { 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_init", 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.960 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Run podman/docker with `--init`", 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.961 INFO:teuthology.orchestra.run.smithi073.stdout: "daemon_cache_timeout": { 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "daemon_cache_timeout", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.962 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "default_cephadm_command_timeout": { 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_cephadm_command_timeout", 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.963 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "900", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.964 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "default_registry": { 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_registry", 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io", 2024-03-09T00:17:26.965 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.966 INFO:teuthology.orchestra.run.smithi073.stdout: "device_cache_timeout": { 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_cache_timeout", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1800", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.967 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache device inventory", 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "device_enhanced_scan": { 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_enhanced_scan", 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.968 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.969 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "facts_cache_timeout": { 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "facts_cache_timeout", 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:26.970 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache host facts data", 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "host_check_interval": { 2024-03-09T00:17:26.971 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "host_check_interval", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to perform a host check", 2024-03-09T00:17:26.972 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "hw_monitoring": { 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hw_monitoring", 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.973 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.974 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "inventory_list_all": { 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "inventory_list_all", 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.975 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:26.976 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.977 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.978 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "log_refresh_metadata": { 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_refresh_metadata", 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.979 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.980 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:26.981 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:26.981 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.981 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.981 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.981 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.982 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:26.983 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:26.984 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:26.985 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:26.986 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.987 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.988 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "*", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.989 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "max_count_per_host": { 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_count_per_host", 2024-03-09T00:17:26.990 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of daemons per service per host", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.991 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "max_osd_draining_count": { 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_osd_draining_count", 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:26.992 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "migration_current": { 2024-03-09T00:17:26.993 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "migration_current", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.994 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "internal - do not modify", 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.995 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "root", 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm-package", 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "root" 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:26.996 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "mode for remote execution of cephadm", 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "oob_default_addr": { 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "oob_default_addr", 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.997 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "169.254.1.1", 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:26.998 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "prometheus_alerts_path": { 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus_alerts_path", 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:26.999 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_insecure": { 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_insecure", 2024-03-09T00:17:27.000 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-09T00:17:27.001 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_password": { 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_password", 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.002 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.003 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_url": { 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_url", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.004 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_username": { 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_username", 2024-03-09T00:17:27.005 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-09T00:17:27.006 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "secure_monitoring_stack": { 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "secure_monitoring_stack", 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.007 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.008 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "service_discovery_port": { 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "service_discovery_port", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8765", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.009 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "cephadm service discovery port", 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "ssh_config_file": { 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssh_config_file", 2024-03-09T00:17:27.010 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.011 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "use_agent": { 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_agent", 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.012 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.013 INFO:teuthology.orchestra.run.smithi073.stdout: "use_repo_digest": { 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_repo_digest", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.014 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_failed_host_check": { 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_failed_host_check", 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.015 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if the host check fails", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.016 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_daemons": { 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_daemons", 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.017 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_hosts": { 2024-03-09T00:17:27.018 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_hosts", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-09T00:17:27.019 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crash", 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.020 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.021 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.022 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.023 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.024 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.025 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.026 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.027 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.028 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "retain_interval": { 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retain_interval", 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.029 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "31536000", 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.030 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_recent_interval": { 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_recent_interval", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1209600", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.031 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.032 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dashboard", 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.033 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.034 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_HOST": { 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.035 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-09T00:17:27.036 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.037 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_ENABLED": { 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_ENABLED", 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.038 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.039 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.040 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "ENABLE_BROWSABLE_API": { 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.041 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.042 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.043 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-09T00:17:27.044 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.045 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.046 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.047 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.048 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-09T00:17:27.049 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.050 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.051 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.052 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.053 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.054 INFO:teuthology.orchestra.run.smithi073.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.055 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_PASSWORD": { 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.056 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.057 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.058 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_URL": { 2024-03-09T00:17:27.059 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_URL", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.060 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_USERNAME": { 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.061 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.062 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.063 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.064 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.065 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.066 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.067 INFO:teuthology.orchestra.run.smithi073.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.068 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "MULTICLUSTER_CONFIG": { 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.069 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "{}", 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.070 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_HOST": { 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.071 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.072 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.073 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-09T00:17:27.074 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.075 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.076 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.077 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.078 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.079 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.080 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.081 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.082 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.083 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.084 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.085 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_ENABLED": { 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.086 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-09T00:17:27.087 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.088 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.089 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.090 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8", 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.091 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-09T00:17:27.092 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "45", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.093 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ACCESS_KEY": { 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.094 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.095 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.096 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SECRET_KEY": { 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-09T00:17:27.097 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.098 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SSL_VERIFY": { 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.099 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.100 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "UNSAFE_TLS_v1_2": { 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.101 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-09T00:17:27.102 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.103 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.104 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.105 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.106 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "cross_origin_url": { 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cross_origin_url", 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.107 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.108 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "crt_file": { 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crt_file", 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.109 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.110 INFO:teuthology.orchestra.run.smithi073.stdout: "debug": { 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "debug", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.111 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable/disable debug options", 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "jwt_token_ttl": { 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "jwt_token_ttl", 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.112 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "28800", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.113 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.114 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.115 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.116 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.117 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.118 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.119 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.120 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.121 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.122 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.123 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "motd": { 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "motd", 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.124 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "The message of the day", 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect_resolve_ip_addr": { 2024-03-09T00:17:27.125 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "redirect_resolve_ip_addr", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.126 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.127 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.128 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8080", 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.129 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:27.130 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.131 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl_server_port": { 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl_server_port", 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.132 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8443", 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.133 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "redirect", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.134 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect" 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.135 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:27.136 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "url_prefix": { 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url_prefix", 2024-03-09T00:17:27.137 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.138 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "devicehealth", 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.139 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_monitoring": { 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_monitoring", 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.140 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "monitor device health metrics", 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.141 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.142 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.143 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.144 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.145 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.146 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.148 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.149 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "mark_out_threshold": { 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mark_out_threshold", 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2419200", 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.150 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_name": { 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_name", 2024-03-09T00:17:27.151 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "device_health_metrics", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.152 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "retention_period": { 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retention_period", 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.153 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15552000", 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain device health metrics", 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.154 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_frequency": { 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_frequency", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.155 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "self_heal": { 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "self_heal", 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.156 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.157 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.158 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and check device health", 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.159 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_threshold": { 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_threshold", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7257600", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.160 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.161 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "diskprediction_local", 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.162 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.163 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.164 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.165 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.166 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.167 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.168 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.169 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "predict_interval": { 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predict_interval", 2024-03-09T00:17:27.170 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.171 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "predictor_model": { 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predictor_model", 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "prophetstor", 2024-03-09T00:17:27.172 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.173 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.174 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.175 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "influx", 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": false, 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "influxdb python module not found", 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "batch_size": { 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "batch_size", 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.176 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5000", 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.177 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "database": { 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "database", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "ceph", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.178 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "hostname": { 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hostname", 2024-03-09T00:17:27.179 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server hostname", 2024-03-09T00:17:27.180 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.181 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "30", 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "5", 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.182 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.183 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.184 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.185 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.186 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.187 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.188 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.189 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.190 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "password": { 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "password", 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.191 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "password of InfluxDB server user", 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "port": { 2024-03-09T00:17:27.192 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "port", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8086", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server port", 2024-03-09T00:17:27.193 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.194 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "false", 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.195 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "threads": { 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threads", 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:27.196 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "32", 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "username": { 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "username", 2024-03-09T00:17:27.197 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "username of InfluxDB server user", 2024-03-09T00:17:27.198 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "verify_ssl": { 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "verify_ssl", 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.199 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "true", 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.200 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "insights", 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.201 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.202 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.203 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.204 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.205 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.206 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.207 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.208 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.209 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "iostat", 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.210 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.211 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.212 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.213 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.214 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.215 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.216 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.217 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.218 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "k8sevents", 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_event_retention_days": { 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ceph_event_retention_days", 2024-03-09T00:17:27.219 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-09T00:17:27.220 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "config_check_secs": { 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_check_secs", 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.221 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "10", 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.222 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.223 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.224 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.225 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.226 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.227 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.228 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.229 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.230 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "localpool", 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.231 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "failure_domain": { 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "failure_domain", 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "host", 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.232 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "failure domain for any created local pool", 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.233 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.234 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.235 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.236 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.237 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.238 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.239 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.240 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.241 INFO:teuthology.orchestra.run.smithi073.stdout: "min_size": { 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_size", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.242 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default min_size for any created local pool", 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "num_rep": { 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "num_rep", 2024-03-09T00:17:27.243 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default replica count for any created local pool", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.244 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_num": { 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_num", 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "128", 2024-03-09T00:17:27.245 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default pg_num for any created local pool", 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.246 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "prefix": { 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prefix", 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.247 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name prefix for any created local pool", 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "subtree": { 2024-03-09T00:17:27.248 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "subtree", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "rack", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.249 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.250 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mds_autoscaler", 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.251 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.252 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.253 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.254 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.255 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.256 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.257 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.258 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirroring", 2024-03-09T00:17:27.259 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.260 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.261 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.262 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.263 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.264 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.265 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.266 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.267 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "nfs", 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.268 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.269 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.270 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.271 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.272 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.273 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.274 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.275 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.276 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.277 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "fail_fs": { 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "fail_fs", 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.278 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.279 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.280 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.281 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.282 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.283 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.284 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.285 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.286 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.287 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator": { 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.288 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm", 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "rook", 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "test_orchestrator" 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Orchestrator backend", 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.289 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_perf_query", 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.290 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.291 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.292 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.293 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.294 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.295 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.296 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.297 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.298 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_support", 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.299 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.300 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.301 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.302 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.303 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.304 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.305 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.306 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.307 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_autoscaler", 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.308 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.309 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.310 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.311 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.312 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.313 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.314 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.315 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.316 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.317 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.318 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "threshold": { 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threshold", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1.0", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.319 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "scaling threshold", 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.320 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "progress", 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_pg_recovery_event": { 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_pg_recovery_event", 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.321 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.322 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.323 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.324 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.325 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.326 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.327 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.328 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.329 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.330 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.331 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.332 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "max_completed_events": { 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_completed_events", 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "50", 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.333 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "number of past completed events to remember", 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.334 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long the module is going to sleep", 2024-03-09T00:17:27.335 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus", 2024-03-09T00:17:27.336 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "cache": { 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cache", 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.337 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.338 INFO:teuthology.orchestra.run.smithi073.stdout: "exclude_perf_counters": { 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "exclude_perf_counters", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.339 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.343 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.344 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.345 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.346 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.347 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.348 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools": { 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools", 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.349 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-09T00:17:27.350 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "300", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.351 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_interval": { 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_interval", 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.352 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15.0", 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.353 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.354 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:27.355 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "9283", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.356 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "stale_cache_strategy": { 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stale_cache_strategy", 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "log", 2024-03-09T00:17:27.357 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.358 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "default", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.359 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "default", 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "error" 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.360 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:27.361 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.362 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_support", 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.363 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.364 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.365 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.366 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.367 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.368 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.369 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.370 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.371 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_snap_create": { 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_snap_create", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.372 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "mirror_snapshot_schedule": { 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirror_snapshot_schedule", 2024-03-09T00:17:27.373 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.374 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "trash_purge_schedule": { 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "trash_purge_schedule", 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.375 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.376 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "restful", 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.377 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_auth": { 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_auth", 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.378 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:27.379 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.380 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.381 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.382 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.383 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.384 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.385 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.386 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.387 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.388 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:27.389 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.390 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.391 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.392 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rgw", 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.393 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.394 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.395 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.396 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.397 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.398 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.399 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.400 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.401 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rook", 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.402 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.403 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.404 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.405 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.406 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.407 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.408 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.409 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.410 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "storage_class": { 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "storage_class", 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.411 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "local", 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.412 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "selftest", 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.413 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.414 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.415 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.416 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.417 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.418 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.419 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.420 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.421 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "roption1": { 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption1", 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.422 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.423 INFO:teuthology.orchestra.run.smithi073.stdout: "roption2": { 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption2", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "xyz", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.424 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption1": { 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption1", 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.425 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.426 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption2": { 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption2", 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.427 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption3": { 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption3", 2024-03-09T00:17:27.428 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.429 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption4": { 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption4", 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.430 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.431 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption5": { 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption5", 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.432 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption6": { 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption6", 2024-03-09T00:17:27.433 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.434 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption7": { 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption7", 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.435 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "42", 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.436 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "testkey": { 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testkey", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.437 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "testlkey": { 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testlkey", 2024-03-09T00:17:27.438 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.439 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "testnewline": { 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testnewline", 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.440 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.441 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snap_schedule", 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_m_granularity": { 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_m_granularity", 2024-03-09T00:17:27.442 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow minute scheduled snapshots", 2024-03-09T00:17:27.443 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "dump_on_update": { 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dump_on_update", 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.444 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "dump database to debug log on update", 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.445 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.446 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.447 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.448 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.449 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.450 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.451 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.452 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.453 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stats", 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.454 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.455 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.456 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.457 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.458 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.459 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.460 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.461 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.462 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.463 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "status", 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.464 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.465 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.466 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.467 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.468 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.469 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.470 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.471 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.472 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telegraf", 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "address": { 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "address", 2024-03-09T00:17:27.473 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.474 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.475 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15", 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.476 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.477 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.478 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.479 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.480 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.481 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.482 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.483 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.484 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telemetry", 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.485 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_basic": { 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_basic", 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.486 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.487 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_crash": { 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_crash", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.488 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_device": { 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_device", 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.489 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.490 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_ident": { 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_ident", 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.491 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.492 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_perf": { 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_perf", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.493 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "contact": { 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "contact", 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.494 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.495 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "description": { 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "description", 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.496 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "device_url": { 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_url", 2024-03-09T00:17:27.497 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.498 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.499 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.500 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "24", 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "8", 2024-03-09T00:17:27.501 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "last_opt_revision": { 2024-03-09T00:17:27.502 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "last_opt_revision", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.503 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard": { 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard", 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.504 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.505 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard_description": { 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard_description", 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.506 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.507 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.508 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.509 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.510 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.511 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.512 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.513 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.514 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.515 INFO:teuthology.orchestra.run.smithi073.stdout: "organization": { 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "organization", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.516 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "proxy": { 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "proxy", 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.517 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.518 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "url": { 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url", 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-09T00:17:27.519 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.520 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "test_orchestrator", 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.521 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.522 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.523 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.524 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.525 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.526 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.527 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.528 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.529 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "volumes", 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.530 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.531 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.532 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.533 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.534 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.535 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.536 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.537 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.538 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_clones": { 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_clones", 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4", 2024-03-09T00:17:27.539 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "periodic_async_work": { 2024-03-09T00:17:27.540 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "periodic_async_work", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.541 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Periodically check for async work", 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_delay": { 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_delay", 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.542 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.543 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_no_wait": { 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_no_wait", 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.544 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.545 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix", 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "discovery_interval": { 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "discovery_interval", 2024-03-09T00:17:27.546 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "100", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.547 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "identifier": { 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "identifier", 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.548 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.549 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.550 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.551 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.552 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.553 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.554 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.555 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.556 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.557 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.558 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.559 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_host": { 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_host", 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.560 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_port": { 2024-03-09T00:17:27.561 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_port", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10051", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.562 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_sender": { 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_sender", 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.563 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.564 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: "modules": [ 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm", 2024-03-09T00:17:27.565 INFO:teuthology.orchestra.run.smithi073.stdout: "dashboard", 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "iostat", 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "nfs", 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "restful", 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "snap_schedule" 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "available_modules": [ 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.566 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "alerts", 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.567 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How frequently to reexamine health status", 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.568 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.569 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.570 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.571 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.572 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.573 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.574 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.575 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.576 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_destination": { 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_destination", 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.577 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email address to send alerts to", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.578 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_from_name": { 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_from_name", 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.579 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "Ceph", 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Email From: name", 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.580 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_host": { 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_host", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.581 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP server", 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_password": { 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_password", 2024-03-09T00:17:27.582 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Password to authenticate with", 2024-03-09T00:17:27.583 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_port": { 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_port", 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.584 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "465", 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP port", 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.585 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_sender": { 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_sender", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.586 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SMTP envelope sender", 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_ssl": { 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_ssl", 2024-03-09T00:17:27.587 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.588 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "smtp_user": { 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "smtp_user", 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.589 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "User to authenticate as", 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.590 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "balancer", 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.591 INFO:teuthology.orchestra.run.smithi073.stdout: "active": { 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "active", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.592 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically balance PGs across cluster", 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_time": { 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_time", 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.593 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0000", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "beginning time of day to automatically balance", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:27.594 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "begin_weekday": { 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "begin_weekday", 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.595 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.596 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_max_iterations": { 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_max_iterations", 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "25", 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:27.597 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "250", 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_metrics": { 2024-03-09T00:17:27.598 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_metrics", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "pgs,objects,bytes", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-09T00:17:27.599 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "crush_compat_step": { 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crush_compat_step", 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.600 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.5", 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0.001", 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "0.999", 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "aggressiveness of optimization", 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.601 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "end_time": { 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_time", 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2359", 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.602 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ending time of day to automatically balance", 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "end_weekday": { 2024-03-09T00:17:27.603 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "end_weekday", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "0", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "6", 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.604 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.605 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.606 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.607 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.607 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.607 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.607 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.607 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.608 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.609 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.610 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.610 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.610 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.610 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.610 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.611 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.612 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.613 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.614 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.615 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "min_score": { 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_score", 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.616 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.617 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "upmap", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.618 INFO:teuthology.orchestra.run.smithi073.stdout: "crush-compat", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "none", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "read", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap-read" 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Balancer mode", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.619 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_ids": { 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_ids", 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.620 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.621 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.622 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_deviation": { 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_deviation", 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.623 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.624 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "upmap_max_optimizations": { 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "upmap_max_optimizations", 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.625 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.626 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm", 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_down_multiplier": { 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_down_multiplier", 2024-03-09T00:17:27.627 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.628 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_refresh_rate": { 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_refresh_rate", 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.629 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "20", 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.630 INFO:teuthology.orchestra.run.smithi073.stdout: "agent_starting_port": { 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "agent_starting_port", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4721", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.631 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_ptrace": { 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_ptrace", 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.632 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-09T00:17:27.633 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_interval": { 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_interval", 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.634 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.635 INFO:teuthology.orchestra.run.smithi073.stdout: "autotune_memory_target_ratio": { 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "autotune_memory_target_ratio", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0.7", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.636 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm_log_destination": { 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cephadm_log_destination", 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.637 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "file", 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "file,syslog", 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: "syslog" 2024-03-09T00:17:27.638 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "cgroups_split": { 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cgroups_split", 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.639 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.640 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "config_checks_enabled": { 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_checks_enabled", 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.641 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.642 INFO:teuthology.orchestra.run.smithi073.stdout: "config_dashboard": { 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_dashboard", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.643 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_alertmanager": { 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_alertmanager", 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.644 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.645 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_base": { 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_base", 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-09T00:17:27.646 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Container image name, without the tag", 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.647 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_elasticsearch": { 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_elasticsearch", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.648 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "elasticsearch container image", 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_grafana": { 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_grafana", 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.649 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.650 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_haproxy": { 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_haproxy", 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-09T00:17:27.651 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "HAproxy container image", 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.652 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_agent": { 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_agent", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.653 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger agent container image", 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_collector": { 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_collector", 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.654 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger collector container image", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.655 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_jaeger_query": { 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_jaeger_query", 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-09T00:17:27.656 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Jaeger query container image", 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_keepalived": { 2024-03-09T00:17:27.657 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_keepalived", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.658 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Keepalived container image", 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_loki": { 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_loki", 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.659 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Loki container image", 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.660 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_node_exporter": { 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_node_exporter", 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.661 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_nvmeof": { 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_nvmeof", 2024-03-09T00:17:27.662 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Nvme-of container image", 2024-03-09T00:17:27.663 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_prometheus": { 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_prometheus", 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.664 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Prometheus container image", 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.665 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_promtail": { 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_promtail", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.666 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Promtail container image", 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "container_image_snmp_gateway": { 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_image_snmp_gateway", 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.667 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "SNMP Gateway container image", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.668 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "container_init": { 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "container_init", 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.669 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Run podman/docker with `--init`", 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.670 INFO:teuthology.orchestra.run.smithi073.stdout: "daemon_cache_timeout": { 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "daemon_cache_timeout", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.671 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "default_cephadm_command_timeout": { 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_cephadm_command_timeout", 2024-03-09T00:17:27.672 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "900", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.673 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "default_registry": { 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "default_registry", 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.674 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "docker.io", 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.675 INFO:teuthology.orchestra.run.smithi073.stdout: "device_cache_timeout": { 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_cache_timeout", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1800", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.676 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache device inventory", 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "device_enhanced_scan": { 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_enhanced_scan", 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.677 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.678 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "facts_cache_timeout": { 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "facts_cache_timeout", 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.679 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "seconds to cache host facts data", 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "host_check_interval": { 2024-03-09T00:17:27.680 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "host_check_interval", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to perform a host check", 2024-03-09T00:17:27.681 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "hw_monitoring": { 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hw_monitoring", 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.682 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.683 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "inventory_list_all": { 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "inventory_list_all", 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.684 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.685 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.686 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.687 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "log_refresh_metadata": { 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_refresh_metadata", 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.688 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.689 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-09T00:17:27.690 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.691 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.692 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.693 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.694 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.695 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.696 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.697 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "*", 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.698 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "max_count_per_host": { 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_count_per_host", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.699 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of daemons per service per host", 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "max_osd_draining_count": { 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_osd_draining_count", 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.700 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.701 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "migration_current": { 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "migration_current", 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.702 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "internal - do not modify", 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.703 INFO:teuthology.orchestra.run.smithi073.stdout: "mode": { 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mode", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "root", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.704 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm-package", 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "root" 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "mode for remote execution of cephadm", 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "oob_default_addr": { 2024-03-09T00:17:27.705 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "oob_default_addr", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "169.254.1.1", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.706 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "prometheus_alerts_path": { 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus_alerts_path", 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.707 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.708 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_insecure": { 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_insecure", 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.709 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_password": { 2024-03-09T00:17:27.710 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_password", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-09T00:17:27.711 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_url": { 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_url", 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.712 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.713 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "registry_username": { 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "registry_username", 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.714 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "secure_monitoring_stack": { 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "secure_monitoring_stack", 2024-03-09T00:17:27.715 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-09T00:17:27.716 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "service_discovery_port": { 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "service_discovery_port", 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.717 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8765", 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "cephadm service discovery port", 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.718 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "ssh_config_file": { 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssh_config_file", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.719 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "use_agent": { 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_agent", 2024-03-09T00:17:27.720 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-09T00:17:27.721 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "use_repo_digest": { 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "use_repo_digest", 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.722 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.723 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_failed_host_check": { 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_failed_host_check", 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.724 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if the host check fails", 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_daemons": { 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_daemons", 2024-03-09T00:17:27.725 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-09T00:17:27.726 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_on_stray_hosts": { 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_on_stray_hosts", 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.727 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.728 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crash", 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.729 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.730 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.731 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.732 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.733 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.734 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.735 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.736 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.737 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "retain_interval": { 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retain_interval", 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "31536000", 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.738 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_recent_interval": { 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_recent_interval", 2024-03-09T00:17:27.739 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1209600", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.740 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dashboard", 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.741 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.742 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_HOST": { 2024-03-09T00:17:27.743 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.744 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.745 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.746 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_ENABLED": { 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_ENABLED", 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.747 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-09T00:17:27.748 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.749 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "ENABLE_BROWSABLE_API": { 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.750 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.751 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.752 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-09T00:17:27.753 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.754 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.755 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.756 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.757 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-09T00:17:27.758 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.759 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.760 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.761 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.762 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-09T00:17:27.763 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.764 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_PASSWORD": { 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.765 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.766 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.767 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_URL": { 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_URL", 2024-03-09T00:17:27.768 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.769 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_API_USERNAME": { 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.770 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.771 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.772 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.773 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.774 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.775 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.776 INFO:teuthology.orchestra.run.smithi073.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.777 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "MULTICLUSTER_CONFIG": { 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.778 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "{}", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.779 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_HOST": { 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.780 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.781 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.782 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-09T00:17:27.783 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.784 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.785 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.786 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.787 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.788 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.789 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.790 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.791 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.792 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.793 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.794 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_ENABLED": { 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.795 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-09T00:17:27.796 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.797 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.798 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.799 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8", 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.800 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-09T00:17:27.801 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "45", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.802 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ACCESS_KEY": { 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.803 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.804 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "admin", 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.805 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SECRET_KEY": { 2024-03-09T00:17:27.806 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.807 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "RGW_API_SSL_VERIFY": { 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.808 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.809 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "UNSAFE_TLS_v1_2": { 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.810 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-09T00:17:27.811 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.812 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.813 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.814 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.815 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "cross_origin_url": { 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cross_origin_url", 2024-03-09T00:17:27.816 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.817 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "crt_file": { 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "crt_file", 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.818 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.819 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "debug": { 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "debug", 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.820 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Enable/disable debug options", 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "jwt_token_ttl": { 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "jwt_token_ttl", 2024-03-09T00:17:27.821 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "28800", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.822 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.823 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.824 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.825 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.826 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.827 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.828 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.829 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.830 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.831 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.832 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "motd": { 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "motd", 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.833 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "The message of the day", 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.834 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect_resolve_ip_addr": { 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "redirect_resolve_ip_addr", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.835 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:27.836 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.837 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.838 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8080", 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.839 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.840 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl_server_port": { 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl_server_port", 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.841 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8443", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.842 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.843 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "redirect", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "redirect" 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.844 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.845 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.846 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "url_prefix": { 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url_prefix", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.847 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.848 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "devicehealth", 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_monitoring": { 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_monitoring", 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.849 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "monitor device health metrics", 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.850 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.851 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.852 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.853 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.854 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.855 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.856 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.857 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.858 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "mark_out_threshold": { 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mark_out_threshold", 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.859 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "2419200", 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.860 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "pool_name": { 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pool_name", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "device_health_metrics", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.861 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "retention_period": { 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "retention_period", 2024-03-09T00:17:27.862 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15552000", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long to retain device health metrics", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.863 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_frequency": { 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_frequency", 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.864 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.865 INFO:teuthology.orchestra.run.smithi073.stdout: "self_heal": { 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "self_heal", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.866 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.867 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how frequently to wake up and check device health", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.868 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "warn_threshold": { 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "warn_threshold", 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7257600", 2024-03-09T00:17:27.869 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.870 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "diskprediction_local", 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.871 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.872 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.873 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.874 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.875 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.876 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.877 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.878 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.879 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "predict_interval": { 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predict_interval", 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "86400", 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.880 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "predictor_model": { 2024-03-09T00:17:27.881 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "predictor_model", 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "prophetstor", 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.882 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.883 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "600", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.884 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.885 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "influx", 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": false, 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "influxdb python module not found", 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "batch_size": { 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "batch_size", 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.886 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5000", 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.887 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "database": { 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "database", 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "ceph", 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.888 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "hostname": { 2024-03-09T00:17:27.889 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "hostname", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server hostname", 2024-03-09T00:17:27.890 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.891 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "30", 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "5", 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.892 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.893 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.894 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.895 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.896 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.897 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.898 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.899 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.900 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "password": { 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "password", 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.901 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "password of InfluxDB server user", 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.902 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "port": { 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "port", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "8086", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.903 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "InfluxDB server port", 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "ssl": { 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ssl", 2024-03-09T00:17:27.904 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "false", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:27.905 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "threads": { 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threads", 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.906 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "32", 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.907 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "username": { 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "username", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.908 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "username of InfluxDB server user", 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "verify_ssl": { 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "verify_ssl", 2024-03-09T00:17:27.909 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "true", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.910 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "insights", 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.911 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.912 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.913 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.914 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.915 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.916 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.917 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.918 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.919 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "iostat", 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.920 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.921 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.922 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.923 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.924 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.925 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.926 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.927 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.928 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "k8sevents", 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.929 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "ceph_event_retention_days": { 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "ceph_event_retention_days", 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "7", 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.930 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "config_check_secs": { 2024-03-09T00:17:27.931 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "config_check_secs", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "10", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-09T00:17:27.932 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.933 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.934 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.935 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.936 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.937 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.938 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.939 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.940 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.941 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "localpool", 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "failure_domain": { 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "failure_domain", 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.942 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "host", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "failure domain for any created local pool", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-09T00:17:27.943 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.944 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.945 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.946 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.947 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.948 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.949 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.950 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.951 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "min_size": { 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "min_size", 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.952 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default min_size for any created local pool", 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.953 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "num_rep": { 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "num_rep", 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3", 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.954 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default replica count for any created local pool", 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_num": { 2024-03-09T00:17:27.955 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_num", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "128", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "default pg_num for any created local pool", 2024-03-09T00:17:27.956 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "prefix": { 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prefix", 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.957 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "name prefix for any created local pool", 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.958 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "subtree": { 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "subtree", 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "rack", 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.959 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.960 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mds_autoscaler", 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.961 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.962 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.963 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.964 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.965 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.966 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.967 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.968 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.969 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirroring", 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.970 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.971 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.972 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.973 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.974 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.975 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.976 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.977 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.978 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "nfs", 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.979 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.980 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.981 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.982 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.983 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.984 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.985 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.986 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.987 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "fail_fs": { 2024-03-09T00:17:27.988 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "fail_fs", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.989 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.990 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.991 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.992 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.993 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:27.994 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:27.995 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.996 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.997 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator": { 2024-03-09T00:17:27.998 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "orchestrator", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:27.999 INFO:teuthology.orchestra.run.smithi073.stdout: "cephadm", 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "rook", 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "test_orchestrator" 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Orchestrator backend", 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.000 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_perf_query", 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.001 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.002 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.003 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.004 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.005 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.006 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.007 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.008 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.009 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "osd_support", 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.010 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.011 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.012 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.013 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.014 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.015 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.016 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.017 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.018 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "pg_autoscaler", 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.019 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.020 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.021 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.022 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.023 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.024 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.025 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.026 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.027 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.028 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.029 INFO:teuthology.orchestra.run.smithi073.stdout: "threshold": { 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "threshold", 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "3.0", 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1.0", 2024-03-09T00:17:28.030 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "scaling threshold", 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.031 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "progress", 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_pg_recovery_event": { 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_pg_recovery_event", 2024-03-09T00:17:28.032 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.033 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.034 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.035 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.036 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.037 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.038 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.039 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.040 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.041 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.042 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.043 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "max_completed_events": { 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_completed_events", 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "50", 2024-03-09T00:17:28.044 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "number of past completed events to remember", 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "sleep_interval": { 2024-03-09T00:17:28.045 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "sleep_interval", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "5", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.046 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "how long the module is going to sleep", 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.047 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "prometheus", 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "cache": { 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "cache", 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.048 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.049 INFO:teuthology.orchestra.run.smithi073.stdout: "exclude_perf_counters": { 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "exclude_perf_counters", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.050 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.051 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.052 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.053 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.054 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.055 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.056 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.057 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.058 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools": { 2024-03-09T00:17:28.059 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.060 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.061 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "300", 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.062 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "scrape_interval": { 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "scrape_interval", 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15.0", 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.063 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:28.064 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "::", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-09T00:17:28.065 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.066 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "9283", 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.067 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "stale_cache_strategy": { 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stale_cache_strategy", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "log", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.068 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_behaviour": { 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_behaviour", 2024-03-09T00:17:28.069 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "default", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "default", 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: "error" 2024-03-09T00:17:28.070 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "standby_error_status_code": { 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "standby_error_status_code", 2024-03-09T00:17:28.071 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "500", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "400", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "599", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.072 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_support", 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.073 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.074 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.075 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.075 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.075 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.075 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.075 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.077 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.078 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.079 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.080 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.081 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.082 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.083 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_snap_create": { 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_snap_create", 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10", 2024-03-09T00:17:28.084 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "mirror_snapshot_schedule": { 2024-03-09T00:17:28.085 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "mirror_snapshot_schedule", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.086 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "trash_purge_schedule": { 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "trash_purge_schedule", 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.087 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.088 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "restful", 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.089 INFO:teuthology.orchestra.run.smithi073.stdout: "enable_auth": { 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enable_auth", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.090 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "key_file": { 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "key_file", 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.091 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.092 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.093 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.094 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.095 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.096 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.097 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.098 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.099 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.100 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "server_addr": { 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_addr", 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.101 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.102 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "server_port": { 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "server_port", 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.103 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.104 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rgw", 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.105 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.106 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.107 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.108 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.109 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.110 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.111 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.112 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.113 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rook", 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.114 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.115 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.116 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.117 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.118 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.119 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.120 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.121 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.122 INFO:teuthology.orchestra.run.smithi073.stdout: "storage_class": { 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "storage_class", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "local", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.123 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.124 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "selftest", 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.125 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.126 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.127 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.128 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.129 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.130 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.131 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.132 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "roption1": { 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption1", 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.133 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.134 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "roption2": { 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "roption2", 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "xyz", 2024-03-09T00:17:28.135 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.136 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption1": { 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption1", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.137 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption2": { 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption2", 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.138 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.139 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption3": { 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption3", 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "float", 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.140 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption4": { 2024-03-09T00:17:28.141 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption4", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.142 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption5": { 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption5", 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.143 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.144 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption6": { 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption6", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.145 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "rwoption7": { 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rwoption7", 2024-03-09T00:17:28.146 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "1", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "42", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.147 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "testkey": { 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testkey", 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.148 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.149 INFO:teuthology.orchestra.run.smithi073.stdout: "testlkey": { 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testlkey", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.150 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "testnewline": { 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "testnewline", 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.151 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.152 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snap_schedule", 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.153 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "allow_m_granularity": { 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "allow_m_granularity", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.154 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "allow minute scheduled snapshots", 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "dump_on_update": { 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "dump_on_update", 2024-03-09T00:17:28.155 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "dump database to debug log on update", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.156 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.157 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.158 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.159 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.160 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.161 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.162 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.163 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.163 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.163 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.163 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 msgr2=0x7f24380a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 0x7f24380a4e30 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f244c069000 tx=0x7f243c00a4c0 comp rx=0 tx=0).stop 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 msgr2=0x7f244c198a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 crypto rx=0x7f2448006010 tx=0x7f244802cd70 comp rx=0 tx=0).stop 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 shutdown_connections 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f244c199550 0x7f244c1a3ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f244c105530 0x7f244c198f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f24380a2970 0x7f24380a4e30 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:28.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 --2- 172.21.15.73:0/709039941 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f244c100460 0x7f244c198a30 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:17:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 >> 172.21.15.73:0/709039941 conn(0x7f244c0772f0 msgr2=0x7f244c0fe490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:17:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.139+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 shutdown_connections 2024-03-09T00:17:28.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:17:26.140+0000 7f2453e32640 1 -- 172.21.15.73:0/709039941 wait complete. 2024-03-09T00:17:28.166 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.166 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.166 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.167 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.168 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "stats", 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.169 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.170 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.171 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.172 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.173 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.174 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.175 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.176 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.177 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "status", 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.178 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.179 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.180 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.181 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.182 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.183 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.184 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.185 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.186 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telegraf", 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "address": { 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "address", 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.187 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.188 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "15", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.189 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.190 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.191 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.192 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.193 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.194 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.195 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.196 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.197 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.198 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "telemetry", 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_basic": { 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_basic", 2024-03-09T00:17:28.199 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.200 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_crash": { 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_crash", 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.201 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.202 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_device": { 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_device", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.203 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_ident": { 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_ident", 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.204 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.205 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "channel_perf": { 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "channel_perf", 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.206 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.207 INFO:teuthology.orchestra.run.smithi073.stdout: "contact": { 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "contact", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.208 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "description": { 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "description", 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.209 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.210 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "device_url": { 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "device_url", 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.211 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "enabled": { 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "enabled", 2024-03-09T00:17:28.212 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.213 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.214 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "24", 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "8", 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.215 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "last_opt_revision": { 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "last_opt_revision", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "1", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.216 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard": { 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard", 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.217 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.218 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "leaderboard_description": { 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "leaderboard_description", 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.219 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.220 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.221 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.222 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.223 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.224 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.225 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.226 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.227 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.228 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "organization": { 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "organization", 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.229 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "proxy": { 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "proxy", 2024-03-09T00:17:28.230 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.231 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "url": { 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "url", 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.232 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.233 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "test_orchestrator", 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.234 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.235 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.236 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.237 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.238 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.239 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.240 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.241 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.242 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "volumes", 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.243 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.244 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.245 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.246 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.247 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.248 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.249 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.250 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.251 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "max_concurrent_clones": { 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "max_concurrent_clones", 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "4", 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.252 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "periodic_async_work": { 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "periodic_async_work", 2024-03-09T00:17:28.253 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Periodically check for async work", 2024-03-09T00:17:28.254 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_delay": { 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_delay", 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.255 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "0", 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.256 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "snapshot_clone_no_wait": { 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snapshot_clone_no_wait", 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "True", 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.257 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.258 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix", 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "can_run": true, 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "error_string": "", 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "module_options": { 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "discovery_interval": { 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "discovery_interval", 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "uint", 2024-03-09T00:17:28.259 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "100", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.260 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "identifier": { 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "identifier", 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.261 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "interval": { 2024-03-09T00:17:28.262 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "interval", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "secs", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "60", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.263 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "log_level": { 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_level", 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.264 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.265 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster": { 2024-03-09T00:17:28.266 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.267 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_cluster_level": { 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_cluster_level", 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.268 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "info", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [ 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "critical", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "debug", 2024-03-09T00:17:28.269 INFO:teuthology.orchestra.run.smithi073.stdout: "error", 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "info", 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "warning" 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.270 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "log_to_file": { 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "log_to_file", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "bool", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 1, 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "False", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.271 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_host": { 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_host", 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.272 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.273 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_port": { 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_port", 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "int", 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "10051", 2024-03-09T00:17:28.274 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.275 INFO:teuthology.orchestra.run.smithi073.stdout: "zabbix_sender": { 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "zabbix_sender", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "str", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "level": "advanced", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "flags": 0, 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "min": "", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "max": "", 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "enum_allowed": [], 2024-03-09T00:17:28.276 INFO:teuthology.orchestra.run.smithi073.stdout: "desc": "", 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: "long_desc": "", 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: "tags": [], 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: "see_also": [] 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.277 INFO:teuthology.orchestra.run.smithi073.stdout: "services": { 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "dashboard": "https://172.21.15.73:8443/" 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "always_on_modules": { 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "octopus": [ 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "balancer", 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "crash", 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "devicehealth", 2024-03-09T00:17:28.278 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_autoscaler", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "progress", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_support", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "status", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "telemetry", 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "volumes" 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.279 INFO:teuthology.orchestra.run.smithi073.stdout: "pacific": [ 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "balancer", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "crash", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "devicehealth", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_autoscaler", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "progress", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_support", 2024-03-09T00:17:28.280 INFO:teuthology.orchestra.run.smithi073.stdout: "status", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "telemetry", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "volumes" 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "quincy": [ 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "balancer", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "crash", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "devicehealth", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator", 2024-03-09T00:17:28.281 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_autoscaler", 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "progress", 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_support", 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "status", 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "telemetry", 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "volumes" 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "reef": [ 2024-03-09T00:17:28.282 INFO:teuthology.orchestra.run.smithi073.stdout: "balancer", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "crash", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "devicehealth", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_autoscaler", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "progress", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_support", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "status", 2024-03-09T00:17:28.283 INFO:teuthology.orchestra.run.smithi073.stdout: "telemetry", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "volumes" 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: ], 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "squid": [ 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "balancer", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "crash", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "devicehealth", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "orchestrator", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "pg_autoscaler", 2024-03-09T00:17:28.284 INFO:teuthology.orchestra.run.smithi073.stdout: "progress", 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "rbd_support", 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "status", 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "telemetry", 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "volumes" 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "last_failure_osd_epoch": 94, 2024-03-09T00:17:28.285 INFO:teuthology.orchestra.run.smithi073.stdout: "active_clients": [ 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "devicehealth", 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 1588172499 2024-03-09T00:17:28.286 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "libcephsqlite", 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.287 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 792499837 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "rbd_support", 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.288 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 2373966919 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snap_schedule", 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.289 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 1755749202 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.290 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "snap_schedule", 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 3657111315 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:28.291 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: "name": "volumes", 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: "addrvec": [ 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: { 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: "type": "v2", 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: "addr": "172.21.15.73:0", 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: "nonce": 490046250 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.292 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:28.293 INFO:teuthology.orchestra.run.smithi073.stdout: ] 2024-03-09T00:17:28.293 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:28.300 DEBUG:tasks.check_counter:Getting stats from a 2024-03-09T00:17:28.300 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-09T00:17:28.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:26 smithi073 ceph-mon[32309]: pgmap v330: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:17:28.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:26 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/555344668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:26 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1742863493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:26 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2771766586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.301 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:26 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/709039941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:26 smithi179 ceph-mon[36637]: pgmap v330: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:26 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/555344668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:26 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1742863493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:26 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2771766586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:26 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/709039941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:26 smithi102 ceph-mon[36106]: pgmap v330: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 KiB/s rd, 100 B/s wr, 2 op/s 2024-03-09T00:17:28.302 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:26 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/555344668' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.303 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:26 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1742863493' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.303 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:26 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2771766586' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:17:28.303 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:26 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/709039941' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-09T00:17:28.498 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:28 smithi073 ceph-mon[32309]: pgmap v331: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:17:28.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:28 smithi102 ceph-mon[36106]: pgmap v331: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:17:28.557 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:17:28.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:28 smithi179 ceph-mon[36637]: pgmap v331: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-09T00:17:30.375 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:30 smithi073 ceph-mon[32309]: pgmap v332: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:30.499 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:30.499 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.001248572, 2024-03-09T00:17:30.500 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.000246940, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:30.501 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:30.502 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 165, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 184000, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.013826717, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.009830267, 2024-03-09T00:17:30.503 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.001694762, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 162, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.009178751, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000056658 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:30.504 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 184000 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 232, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 219, 2024-03-09T00:17:30.505 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 1579408, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 130384, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.045293115, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.014958182, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.034859659, 2024-03-09T00:17:30.506 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.004431981, 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 219, 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.012307376, 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000056198 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:30.507 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 1579408, 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 130384 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: "cct": { 2024-03-09T00:17:30.508 INFO:teuthology.orchestra.run.smithi073.stdout: "total_workers": 0, 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "unhealthy_workers": 0 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "mempool": { 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:30.509 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:30.510 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:30.511 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:30.512 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_items": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:30.513 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_bytes": 149017, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_items": 31, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_bytes": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_items": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:30.514 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_items": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:30.515 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_items": 0, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_items": 2, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_items": 0, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_items": 0 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:30.516 INFO:teuthology.orchestra.run.smithi073.stdout: "val": 0, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "max": 104857600, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "get_started": 0, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "get": 230, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "get_sum": 1544273, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_success": 230, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "take": 0, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "take_sum": 0, 2024-03-09T00:17:30.517 INFO:teuthology.orchestra.run.smithi073.stdout: "put": 230, 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: "put_sum": 1544273, 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: "wait": { 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:30.518 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:30.519 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:30 smithi102 ceph-mon[36106]: pgmap v332: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:30.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:30 smithi179 ceph-mon[36637]: pgmap v332: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:31.157 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.a 2024-03-09T00:17:31.157 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.a 2024-03-09T00:17:31.157 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.a 2024-03-09T00:17:31.158 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.a 2024-03-09T00:17:31.158 DEBUG:tasks.check_counter:Getting stats from d 2024-03-09T00:17:31.158 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-09T00:17:31.400 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:17:32.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:32 smithi102 ceph-mon[36106]: pgmap v333: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:32.564 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:32 smithi073 ceph-mon[32309]: pgmap v333: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:32.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:32 smithi179 ceph-mon[36637]: pgmap v333: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:33.158 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:33.158 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:33.158 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 233, 2024-03-09T00:17:33.158 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 222, 2024-03-09T00:17:33.158 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 1577776, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 133056, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.050184401, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.016315901, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.037821685, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.004520240, 2024-03-09T00:17:33.159 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 221, 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.010839209, 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000049046 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:33.160 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 1577776, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 133056 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 170, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 184880, 2024-03-09T00:17:33.161 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.014493894, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.010371243, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.001724529, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 167, 2024-03-09T00:17:33.162 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.014109053, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000084485 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:33.163 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 184880 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 4, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 4, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 2992, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 448, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:33.164 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.000648558, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.000036664, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.000604063, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 3, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000705695, 2024-03-09T00:17:33.165 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000235231 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.166 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 448 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "cct": { 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "total_workers": 0, 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "unhealthy_workers": 0 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "mempool": { 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:33.167 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:33.168 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:33.169 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:33.170 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_items": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:33.171 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_bytes": 148731, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_items": 31, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_bytes": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_items": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:33.172 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:33.173 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:33.173 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:33.173 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_items": 0, 2024-03-09T00:17:33.173 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_items": 0, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_items": 2, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:33.174 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_items": 0, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_items": 0 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "val": 0, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "max": 104857600, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "get_started": 0, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "get": 235, 2024-03-09T00:17:33.175 INFO:teuthology.orchestra.run.smithi073.stdout: "get_sum": 1544923, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_success": 235, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "take": 0, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "take_sum": 0, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "put": 235, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "put_sum": 1544923, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "wait": { 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:33.176 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:33.177 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:33.177 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:33.177 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:33.177 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:33.895 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.d 2024-03-09T00:17:33.896 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.d 2024-03-09T00:17:33.896 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.d 2024-03-09T00:17:33.896 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.d 2024-03-09T00:17:33.896 DEBUG:tasks.check_counter:Getting stats from g 2024-03-09T00:17:33.896 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.g perf dump 2024-03-09T00:17:34.141 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:17:34.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:34 smithi102 ceph-mon[36106]: pgmap v334: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:34.536 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:34 smithi073 ceph-mon[32309]: pgmap v334: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:34.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:34 smithi179 ceph-mon[36637]: pgmap v334: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:35.943 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:35.943 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:35.943 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:35.944 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.001108402, 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.000256452, 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:35.945 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:35.946 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:35.946 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.946 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:35.946 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:35.946 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:35.947 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 167, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 184352, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.013915196, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.009735945, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.001742278, 2024-03-09T00:17:35.948 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 164, 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.009404964, 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000057347 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:35.949 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 184352 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:35.950 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 234, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 221, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 1579952, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 131632, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.043286554, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.014759057, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.032753313, 2024-03-09T00:17:35.951 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.004561260, 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 221, 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.011625430, 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000052603 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:35.952 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 1579952, 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 131632 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "cct": { 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "total_workers": 0, 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: "unhealthy_workers": 0 2024-03-09T00:17:35.953 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "mempool": { 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:35.954 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:35.955 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:35.956 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_items": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:35.957 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_bytes": 149057, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_items": 32, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_bytes": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_items": 0, 2024-03-09T00:17:35.958 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_items": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_items": 0, 2024-03-09T00:17:35.959 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_items": 2, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_items": 0, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_items": 0 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "val": 0, 2024-03-09T00:17:35.960 INFO:teuthology.orchestra.run.smithi073.stdout: "max": 104857600, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "get_started": 0, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "get": 232, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "get_sum": 1544533, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_success": 232, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "take": 0, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "take_sum": 0, 2024-03-09T00:17:35.961 INFO:teuthology.orchestra.run.smithi073.stdout: "put": 232, 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: "put_sum": 1544533, 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: "wait": { 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:35.962 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:36.405 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:36 smithi073 ceph-mon[32309]: pgmap v335: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:36.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:36 smithi102 ceph-mon[36106]: pgmap v335: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:36.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:36 smithi179 ceph-mon[36637]: pgmap v335: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:36.769 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.g 2024-03-09T00:17:36.769 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.g 2024-03-09T00:17:36.769 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.g 2024-03-09T00:17:36.769 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.g 2024-03-09T00:17:36.770 DEBUG:tasks.check_counter:Getting stats from j 2024-03-09T00:17:36.770 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.j perf dump 2024-03-09T00:17:37.014 INFO:teuthology.orchestra.run.smithi073.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.a/config 2024-03-09T00:17:38.511 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:38 smithi073 ceph-mon[32309]: pgmap v336: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:38.512 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:38 smithi102 ceph-mon[36106]: pgmap v336: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:38.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:38 smithi179 ceph-mon[36637]: pgmap v336: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:38.793 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-03-09T00:17:38.793 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:38.793 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 4, 2024-03-09T00:17:38.793 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 4, 2024-03-09T00:17:38.793 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 2992, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 448, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.001597230, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.000036893, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.000835427, 2024-03-09T00:17:38.794 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 3, 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000938689, 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000312896 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:38.795 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 448 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.796 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 235, 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 224, 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 1578320, 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 134304, 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:38.797 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.045010216, 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.016100412, 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.032596233, 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.004637859, 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:38.798 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 223, 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.011831375, 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000053055 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:38.799 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 1578320, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 134304 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages": 172, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:38.800 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_bytes": 185232, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_total_time": 0.015796505, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_send_time": 0.011198513, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_recv_time": 0.002004384, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:38.801 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 169, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.010104408, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000059789 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.802 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: "msgr_send_encrypted_bytes": 185232 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: "cct": { 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: "total_workers": 0, 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: "unhealthy_workers": 0 2024-03-09T00:17:38.803 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "mempool": { 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:38.804 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:38.805 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:38.806 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_items": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:38.807 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_bytes": 148754, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_anon_items": 31, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_bytes": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_items": 0, 2024-03-09T00:17:38.808 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_items": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:38.809 INFO:teuthology.orchestra.run.smithi073.stdout: "pgmap_items": 0, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "mds_co_items": 2, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_1_items": 0, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "unittest_2_items": 0 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: }, 2024-03-09T00:17:38.810 INFO:teuthology.orchestra.run.smithi073.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "val": 0, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "max": 104857600, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "get_started": 0, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "get": 237, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "get_sum": 1545183, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "get_or_fail_success": 237, 2024-03-09T00:17:38.811 INFO:teuthology.orchestra.run.smithi073.stdout: "take": 0, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "take_sum": 0, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "put": 237, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "put_sum": 1545183, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "wait": { 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "avgcount": 0, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "sum": 0.000000000, 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: "avgtime": 0.000000000 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:38.812 INFO:teuthology.orchestra.run.smithi073.stdout: } 2024-03-09T00:17:38.813 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-03-09T00:17:39.505 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.j 2024-03-09T00:17:39.505 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.j 2024-03-09T00:17:39.505 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.j 2024-03-09T00:17:39.505 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.j 2024-03-09T00:17:39.505 DEBUG:tasks.check_counter:Getting stats from b 2024-03-09T00:17:39.506 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.b perf dump 2024-03-09T00:17:39.751 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:17:40.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:40 smithi102 ceph-mon[36106]: pgmap v337: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:40.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:40 smithi179 ceph-mon[36637]: pgmap v337: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:40.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:40 smithi073 ceph-mon[32309]: pgmap v337: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:41.527 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-09T00:17:41.527 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:41.527 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 32437, 2024-03-09T00:17:41.527 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 31831, 2024-03-09T00:17:41.527 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 17358059, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 31582012, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 10, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 5, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 1.510889856, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.506018983, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.364496978, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.529899478, 2024-03-09T00:17:41.528 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 31829, 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 4.585716849, 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000144073 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 930, 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000152575, 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000164 2024-03-09T00:17:41.529 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 337904, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 99232 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 22251, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 22152, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 13286058, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 38118197, 2024-03-09T00:17:41.530 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 183, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 5, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 1.193165525, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.426784522, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.330164061, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.373328953, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 22152, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 5.081323054, 2024-03-09T00:17:41.531 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000229384 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 282, 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000055060, 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000195 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1672736, 2024-03-09T00:17:41.532 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 170512 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 46799, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 63172, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 20298777, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 31292833, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 18, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 5, 2024-03-09T00:17:41.533 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 2.186987972, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.913336599, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 2.095989932, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.431356286, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 63169, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 4.789153721, 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000075814 2024-03-09T00:17:41.534 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 34851, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.006511878, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000186 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 98720, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 495200 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.535 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 1, 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "finisher-MDSRank": { 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "queue_len": 0, 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "complete_latency": { 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 43262, 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 19.041832634, 2024-03-09T00:17:41.536 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000440151 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "finisher-PurgeQueue": { 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "queue_len": 0, 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "complete_latency": { 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 3888, 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 1.275464300, 2024-03-09T00:17:41.537 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000328051 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "mds": { 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "request": 10742, 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "reply": 10742, 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "reply_latency": { 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 10742, 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 440.187998595, 2024-03-09T00:17:41.538 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.040978216 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "slow_reply": 0, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "forward": 0, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_fetch_complete": 0, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_fetch_keys": 0, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_commit": 617, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_split": 25, 2024-03-09T00:17:41.539 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_merge": 2, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes": 5137, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_top": 976, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_bottom": 4161, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_pin_tail": 0, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_pinned": 3635, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_expired": 0, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "inodes_with_caps": 3520, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "caps": 3521, 2024-03-09T00:17:41.540 INFO:teuthology.orchestra.run.smithi102.stdout: "subtrees": 2, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse": 10989, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_hit": 10462, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_forward": 0, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_discover": 0, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_dir_fetch": 0, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_remote_ino": 0, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "traverse_lock": 11, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "load_cent": 0, 2024-03-09T00:17:41.541 INFO:teuthology.orchestra.run.smithi102.stdout: "q": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "exported": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "exported_inodes": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "imported": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "imported_inodes": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "openino_dir_fetch": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "openino_backtrace_fetch": 57278, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "openino_peer_discover": 0, 2024-03-09T00:17:41.542 INFO:teuthology.orchestra.run.smithi102.stdout: "root_rfiles": 3452, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "root_rbytes": 26509593, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "root_rsnaps": 6, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_backtrace_fetch": 57278, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_set_tag": 0, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_backtrace_repaired": 0, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_inotable_repaired": 0, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_dir_inodes": 1284, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_dir_base_inodes": 20, 2024-03-09T00:17:41.543 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_dirfrag_rstats": 1264, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "scrub_file_inodes": 55994, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_inode_file_caps": 0, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_cap_op_revoke": 273, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_cap_op_grant": 11018, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_cap_op_trunc": 8, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_client_caps": 10734, 2024-03-09T00:17:41.544 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_client_caps_dirty": 6562, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_client_cap_release": 1003, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "process_request_cap_release": 3505 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_cache": { 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_update": 0, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_update_receipt": 0, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_try_discover": 0, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_send_discover": 0, 2024-03-09T00:17:41.545 INFO:teuthology.orchestra.run.smithi102.stdout: "dir_handle_discover": 0, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "num_strays": 0, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "num_strays_delayed": 0, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "num_strays_enqueuing": 0, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "strays_created": 3217, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "strays_enqueued": 3215, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "strays_reintegrated": 2, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "strays_migrated": 0, 2024-03-09T00:17:41.546 INFO:teuthology.orchestra.run.smithi102.stdout: "num_recovering_processing": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "num_recovering_enqueued": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "num_recovering_prioritized": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "recovery_started": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "recovery_completed": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_enqueue_scrub": 20, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_exportdir": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_flush": 0, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_fragmentdir": 27, 2024-03-09T00:17:41.547 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_fragstats": 0, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "ireq_inodestats": 0 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_log": { 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "evlrg": 0, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "evadd": 23701, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "evex": 20181, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "evtrm": 20181, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "ev": 3520, 2024-03-09T00:17:41.548 INFO:teuthology.orchestra.run.smithi102.stdout: "evexg": 0, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "evexd": 15810, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segadd": 40, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segex": 34, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segtrm": 34, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "seg": 6, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segmjr": 1, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segexg": 0, 2024-03-09T00:17:41.549 INFO:teuthology.orchestra.run.smithi102.stdout: "segexd": 22, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "expos": 47532541, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "wrpos": 54834278, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "rdpos": 0, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "jlat": { 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 495, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 17.744766912, 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.035848013 2024-03-09T00:17:41.550 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "replayed": 0 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_mem": { 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "ino": 3540, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "ino+": 6775, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "ino-": 3235, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "dir": 248, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "dir+": 299, 2024-03-09T00:17:41.551 INFO:teuthology.orchestra.run.smithi102.stdout: "dir-": 51, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "dn": 5137, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "dn+": 8377, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "dn-": 3240, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "cap": 3521, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "cap+": 6752, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "cap-": 3231, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "rss": 109552, 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: "heap": 215420 2024-03-09T00:17:41.552 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_server": { 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "dispatch_client_request": 11017, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "dispatch_server_request": 0, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_client_request": 10742, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_client_session": 37, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "handle_peer_request": 0, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "req_create_latency": { 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 6650, 2024-03-09T00:17:41.553 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 424.353917129, 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.063812619 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "req_getattr_latency": { 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 125, 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.020395341, 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000163162 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.554 INFO:teuthology.orchestra.run.smithi102.stdout: "req_getfilelock_latency": { 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "req_link_latency": { 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 2, 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.004032776, 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.002016388 2024-03-09T00:17:41.555 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookup_latency": { 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 9, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.005377725, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000597525 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookuphash_latency": { 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.556 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookupino_latency": { 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 2, 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000328940, 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000164470 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookupname_latency": { 2024-03-09T00:17:41.557 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookupparent_latency": { 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.558 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lookupsnap_latency": { 2024-03-09T00:17:41.559 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.559 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.559 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.559 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.559 INFO:teuthology.orchestra.run.smithi102.stdout: "req_lssnap_latency": { 2024-03-09T00:17:41.561 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 11, 2024-03-09T00:17:41.561 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.001493810, 2024-03-09T00:17:41.561 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000135800 2024-03-09T00:17:41.561 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "req_mkdir_latency": { 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 85, 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 2.120558406, 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.024947745 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "req_mknod_latency": { 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.562 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "req_mksnap_latency": { 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 11, 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.062102457, 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.005645677 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "req_open_latency": { 2024-03-09T00:17:41.563 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 441, 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.948873244, 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.002151640 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "req_readdir_latency": { 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 107, 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.534230801, 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.004992811 2024-03-09T00:17:41.564 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "req_rename_latency": { 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 81, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.190609510, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.002353203 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "req_renamesnap_latency": { 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.565 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "req_snapdiff_latency": { 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "req_rmdir_latency": { 2024-03-09T00:17:41.566 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 22, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 1.054473547, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.047930615 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "req_rmsnap_latency": { 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 5, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.019702231, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.003940446 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.567 INFO:teuthology.orchestra.run.smithi102.stdout: "req_rmxattr_latency": { 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "req_setattr_latency": { 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 10, 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.072638942, 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.007263894 2024-03-09T00:17:41.568 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "req_setdirlayout_latency": { 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "req_setfilelock_latency": { 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 8, 2024-03-09T00:17:41.569 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.003456372, 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000432046 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "req_setlayout_latency": { 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "req_setxattr_latency": { 2024-03-09T00:17:41.570 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 8, 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.020559049, 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.002569881 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "req_symlink_latency": { 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 7, 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.004489671, 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000641381 2024-03-09T00:17:41.571 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "req_unlink_latency": { 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 3158, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 10.770758644, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.003410626 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "cap_revoke_eviction": 0, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "cap_acquisition_throttle": 0, 2024-03-09T00:17:41.572 INFO:teuthology.orchestra.run.smithi102.stdout: "req_getvxattr_latency": { 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_sessions": { 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "session_count": 2, 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "session_add": 6, 2024-03-09T00:17:41.573 INFO:teuthology.orchestra.run.smithi102.stdout: "session_remove": 4, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "sessions_open": 2, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "sessions_stale": 0, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "total_load": 56, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "average_load": 28, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "avg_session_uptime": 650, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mdthresh_evicted": 0 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.574 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:41.575 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:41.576 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:41.577 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:41.578 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 428910, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 3647, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 1584, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 18, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:41.579 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 9608, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 184, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-03-09T00:17:41.580 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 13627416, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 159148, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "objecter": { 2024-03-09T00:17:41.581 INFO:teuthology.orchestra.run.smithi102.stdout: "op_active": 0, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_laggy": 0, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_send": 77762, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_send_bytes": 63500536, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_resend": 37, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_reply": 77762, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "op_latency": { 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 77762, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 4442.510600563, 2024-03-09T00:17:41.582 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.057129582 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "op_inflight": 0, 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "oplen_avg": { 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 77762, 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 97273 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "op": 77762, 2024-03-09T00:17:41.583 INFO:teuthology.orchestra.run.smithi102.stdout: "op_r": 57350, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "op_w": 20412, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "op_rmw": 0, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "op_pg": 0, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_stat": 328, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_create": 12135, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_read": 52, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_write": 527, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_writefull": 67, 2024-03-09T00:17:41.584 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_writesame": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_append": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_zero": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_truncate": 9, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_delete": 6981, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_mapext": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_sparse_read": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_clonerange": 0, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_getxattr": 57298, 2024-03-09T00:17:41.585 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_setxattr": 18334, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_cmpxattr": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_rmxattr": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_resetxattrs": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_call": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_watch": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_notify": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_src_cmpxattr": 0, 2024-03-09T00:17:41.586 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_pgls": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_pgls_filter": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "osdop_other": 8, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "linger_active": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "linger_send": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "linger_resend": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "linger_ping": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "poolop_active": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "poolop_send": 0, 2024-03-09T00:17:41.587 INFO:teuthology.orchestra.run.smithi102.stdout: "poolop_resend": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "poolstat_active": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "poolstat_send": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "poolstat_resend": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "statfs_active": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "statfs_send": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "statfs_resend": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "command_active": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "command_send": 0, 2024-03-09T00:17:41.588 INFO:teuthology.orchestra.run.smithi102.stdout: "command_resend": 0, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "map_epoch": 104, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "map_full": 0, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "map_inc": 20, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_sessions": 12, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_session_open": 12, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_session_close": 0, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_laggy": 0, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_wr": 1303, 2024-03-09T00:17:41.589 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_rd": 0, 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_del": 231 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "oft": { 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_total_objs": 1, 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_total_kv_pairs": 0, 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_total_updates": 0, 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: "omap_total_removes": 0 2024-03-09T00:17:41.590 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "osd-slow-ops": { 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "slow_ops_count": 0 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "purge_queue": { 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executing_ops": 0, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executing_ops_high_water": 49, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executing": 0, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executing_high_water": 26, 2024-03-09T00:17:41.591 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executed_ops": 6415, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_executed": 3215, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "pq_item_in_journal": 0 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 101483, 2024-03-09T00:17:41.592 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 42092383, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 101483, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 101483, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 42092383, 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:41.593 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-objecter_bytes": { 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-03-09T00:17:41.594 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 0, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 0, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 77762, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 63850851, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 77762, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 63850851, 2024-03-09T00:17:41.595 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-objecter_ops": { 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 1024, 2024-03-09T00:17:41.596 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 0, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 0, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 77762, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 77762, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 77762, 2024-03-09T00:17:41.597 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 77762, 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-write_buf_throttle": { 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:41.598 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 3758096384, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 3215, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 350315, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 3215, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 32, 2024-03-09T00:17:41.599 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 350315, 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-write_buf_throttle-0x56270bf8c6e0": { 2024-03-09T00:17:41.600 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 3758096384, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 23701, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 50639974, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 23701, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:41.601 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 495, 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 50639974, 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:41.602 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-09T00:17:42.263 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:42 smithi102 ceph-mon[36106]: pgmap v338: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:42.384 INFO:tasks.check_counter:Daemon mds.b mds.root_rsnaps=6 2024-03-09T00:17:42.385 INFO:tasks.check_counter:Daemon mds.b mds_server.req_mksnap_latency.avgcount=11 2024-03-09T00:17:42.385 INFO:tasks.check_counter:Daemon mds.b mds_server.req_rmsnap_latency.avgcount=5 2024-03-09T00:17:42.385 INFO:tasks.check_counter:Daemon mds.b mds.dir_split=25 2024-03-09T00:17:42.385 DEBUG:tasks.check_counter:Getting stats from e 2024-03-09T00:17:42.385 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.e perf dump 2024-03-09T00:17:42.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:42 smithi179 ceph-mon[36637]: pgmap v338: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:42.632 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:17:42.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:42 smithi073 ceph-mon[32309]: pgmap v338: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:44.422 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-09T00:17:44.422 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:44.422 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:44.422 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.001052649, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000202936, 2024-03-09T00:17:44.423 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:44.424 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 6, 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 174, 2024-03-09T00:17:44.425 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 3312, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 185680, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.011164774, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.007354842, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.001700097, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:44.426 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 172, 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.018804678, 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000109329 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:44.427 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 185680 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 235, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 224, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1578320, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 134304, 2024-03-09T00:17:44.428 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.048046563, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.014424018, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.036895667, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.005573008, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 223, 2024-03-09T00:17:44.429 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.011644762, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000052218 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1578320, 2024-03-09T00:17:44.430 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 134304 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:44.431 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:44.432 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:44.433 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:44.434 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:44.435 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 148714, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 30, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:44.436 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:44.437 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-03-09T00:17:44.438 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 237, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1545183, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 237, 2024-03-09T00:17:44.439 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 237, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1545183, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:44.440 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:44.441 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-09T00:17:44.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:44 smithi102 ceph-mon[36106]: pgmap v339: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:44.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:44 smithi179 ceph-mon[36637]: pgmap v339: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:44.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:44 smithi073 ceph-mon[32309]: pgmap v339: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:45.362 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.e 2024-03-09T00:17:45.362 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.e 2024-03-09T00:17:45.363 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.e 2024-03-09T00:17:45.363 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.e 2024-03-09T00:17:45.363 DEBUG:tasks.check_counter:Getting stats from h 2024-03-09T00:17:45.363 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.h perf dump 2024-03-09T00:17:45.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:45 smithi102 ceph-mon[36106]: pgmap v340: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:45.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:45 smithi179 ceph-mon[36637]: pgmap v340: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:45.603 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:17:45.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:45 smithi073 ceph-mon[32309]: pgmap v340: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:47.330 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-09T00:17:47.330 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:47.330 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:47.330 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.001062372, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000183787, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:47.331 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:47.332 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 234, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 223, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1578048, 2024-03-09T00:17:47.333 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 133680, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.047543991, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.013961694, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.035393085, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.006004405, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 222, 2024-03-09T00:17:47.334 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.011461550, 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000051628 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.335 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1578048, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 133680 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 6, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 173, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 3312, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 185504, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:47.336 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.014598046, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.008008578, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.004359610, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 171, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.016063967, 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000093941 2024-03-09T00:17:47.337 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 185504 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.338 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:47.339 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:47.340 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:47.341 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:47.342 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 148977, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 30, 2024-03-09T00:17:47.343 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:47.344 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-03-09T00:17:47.345 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 236, 2024-03-09T00:17:47.346 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1545053, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 236, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 236, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1545053, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:47.347 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:47.348 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:47.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:47.348 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:47.348 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-09T00:17:47.348 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:47 smithi102 ceph-mon[36106]: pgmap v341: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:47.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:47 smithi179 ceph-mon[36637]: pgmap v341: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:47.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:47 smithi073 ceph-mon[32309]: pgmap v341: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:48.065 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.h 2024-03-09T00:17:48.065 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.h 2024-03-09T00:17:48.065 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.h 2024-03-09T00:17:48.065 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.h 2024-03-09T00:17:48.065 DEBUG:tasks.check_counter:Getting stats from k 2024-03-09T00:17:48.066 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.k perf dump 2024-03-09T00:17:48.303 INFO:teuthology.orchestra.run.smithi102.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.b/config 2024-03-09T00:17:49.455 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:49 smithi102 ceph-mon[36106]: pgmap v342: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:49.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:49 smithi179 ceph-mon[36637]: pgmap v342: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:49.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:49 smithi073 ceph-mon[32309]: pgmap v342: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout:{ 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 239, 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 228, 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 1579408, 2024-03-09T00:17:50.050 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 136800, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.052383787, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.015365501, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.039270505, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.005919023, 2024-03-09T00:17:50.051 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 227, 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.012711794, 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000055999 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:50.052 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 1579408, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 136800 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 174, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:50.053 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 185584, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.011747898, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.007736678, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.001657024, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:50.054 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 171, 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.019598005, 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000114608 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:50.055 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 185584 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_messages": 4, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages": 4, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_bytes": 2992, 2024-03-09T00:17:50.056 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_bytes": 448, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_total_time": 0.000711377, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_send_time": 0.000033135, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_recv_time": 0.000675612, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 3, 2024-03-09T00:17:50.057 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000674250, 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000224750 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.058 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_recv_encrypted_bytes": 2992, 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "msgr_send_encrypted_bytes": 448 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "cct": { 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "total_workers": 0, 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "unhealthy_workers": 0 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "mempool": { 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:50.059 INFO:teuthology.orchestra.run.smithi102.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:50.060 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:50.061 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:50.062 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_items": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:50.063 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_bytes": 148714, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_anon_items": 30, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_bytes": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_items": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:50.064 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_items": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:50.065 INFO:teuthology.orchestra.run.smithi102.stdout: "pgmap_items": 0, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "mds_co_items": 2, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_1_items": 0, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "unittest_2_items": 0 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: }, 2024-03-09T00:17:50.066 INFO:teuthology.orchestra.run.smithi102.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "val": 0, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "max": 104857600, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "get_started": 0, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "get": 241, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "get_sum": 1545703, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "get_or_fail_success": 241, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "take": 0, 2024-03-09T00:17:50.067 INFO:teuthology.orchestra.run.smithi102.stdout: "take_sum": 0, 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "put": 241, 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "put_sum": 1545703, 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "wait": { 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "avgcount": 0, 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "sum": 0.000000000, 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: "avgtime": 0.000000000 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:50.068 INFO:teuthology.orchestra.run.smithi102.stdout: } 2024-03-09T00:17:50.069 INFO:teuthology.orchestra.run.smithi102.stdout:} 2024-03-09T00:17:50.867 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.k 2024-03-09T00:17:50.867 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.k 2024-03-09T00:17:50.867 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.k 2024-03-09T00:17:50.867 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.k 2024-03-09T00:17:50.867 DEBUG:tasks.check_counter:Getting stats from c 2024-03-09T00:17:50.868 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.c perf dump 2024-03-09T00:17:51.117 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:17:51.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:51 smithi102 ceph-mon[36106]: pgmap v343: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:51.528 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:51 smithi179 ceph-mon[36637]: pgmap v343: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:51.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:51 smithi073 ceph-mon[32309]: pgmap v343: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:52.868 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:52.869 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.001132104, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.000273278, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:52.870 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:52.871 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 2, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 172, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 185232, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:52.872 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.012033611, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.007807625, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.001713687, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 169, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.015820992, 2024-03-09T00:17:52.873 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000093615 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 185232 2024-03-09T00:17:52.874 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 240, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 227, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 1581584, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 135376, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.048894701, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.014814886, 2024-03-09T00:17:52.875 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.037521117, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.005862005, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 227, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.013711575, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000060403 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:52.876 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 1581584, 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 135376 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "cct": { 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "total_workers": 0, 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "unhealthy_workers": 0 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.877 INFO:teuthology.orchestra.run.smithi179.stdout: "mempool": { 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:52.878 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:52.879 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:52.880 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_items": 0, 2024-03-09T00:17:52.881 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_bytes": 148977, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_items": 30, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:52.882 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bytes": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_items": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_items": 0, 2024-03-09T00:17:52.883 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_items": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_items": 2, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_items": 0, 2024-03-09T00:17:52.884 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_items": 0 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 104857600, 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:52.885 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 238, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 1545313, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 238, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 238, 2024-03-09T00:17:52.886 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 1545313, 2024-03-09T00:17:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:52.887 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:52.888 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:52.888 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-09T00:17:53.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:53 smithi102 ceph-mon[36106]: pgmap v344: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:53.524 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:53 smithi179 ceph-mon[36637]: pgmap v344: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:53.550 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.c 2024-03-09T00:17:53.550 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.c 2024-03-09T00:17:53.550 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.c 2024-03-09T00:17:53.551 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.c 2024-03-09T00:17:53.551 DEBUG:tasks.check_counter:Getting stats from f 2024-03-09T00:17:53.551 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.f perf dump 2024-03-09T00:17:53.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:53 smithi073 ceph-mon[32309]: pgmap v344: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:53.797 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:17:55.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:55 smithi102 ceph-mon[36106]: pgmap v345: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:55.528 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:55 smithi179 ceph-mon[36637]: pgmap v345: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 238, 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 227, 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 1579136, 2024-03-09T00:17:55.651 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 136176, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 4, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 0, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.051976016, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.015811892, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.037988112, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.006179426, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 226, 2024-03-09T00:17:55.652 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.013295382, 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000058829 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.653 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 1579136, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 136176 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 6, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 175, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 3312, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 185856, 2024-03-09T00:17:55.654 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 3, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.011813618, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.007684056, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.001951703, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 173, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.016467902, 2024-03-09T00:17:55.655 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000095190 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 3312, 2024-03-09T00:17:55.656 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 185856 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 0, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 2, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 0, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 2, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 1, 2024-03-09T00:17:55.657 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.000303366, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.000232527, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:55.658 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.659 INFO:teuthology.orchestra.run.smithi179.stdout: "cct": { 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "total_workers": 0, 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "unhealthy_workers": 0 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "mempool": { 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:55.660 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:55.661 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:55.662 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:55.663 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:55.663 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:55.663 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:55.663 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:55.663 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:55.664 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:55.664 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:55.664 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:55.664 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:55.664 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:55.665 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:55.665 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_items": 0, 2024-03-09T00:17:55.665 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:55.665 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:55.666 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:55.666 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:55.666 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_bytes": 148731, 2024-03-09T00:17:55.666 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_items": 31, 2024-03-09T00:17:55.667 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:55.667 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:55.667 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bytes": 0, 2024-03-09T00:17:55.667 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_items": 0, 2024-03-09T00:17:55.667 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:55.668 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:55.668 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:55.668 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:55.668 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_bytes": 0, 2024-03-09T00:17:55.669 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_items": 0, 2024-03-09T00:17:55.669 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:55.669 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:55.669 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:55.669 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_items": 0, 2024-03-09T00:17:55.670 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_bytes": 928, 2024-03-09T00:17:55.670 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_items": 2, 2024-03-09T00:17:55.670 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:55.670 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_items": 0, 2024-03-09T00:17:55.671 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:55.671 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_items": 0 2024-03-09T00:17:55.671 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:55.671 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:55.672 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:55.672 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 104857600, 2024-03-09T00:17:55.672 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:55.672 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 240, 2024-03-09T00:17:55.672 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 1545573, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 240, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 240, 2024-03-09T00:17:55.673 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 1545573, 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:55.674 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-09T00:17:55.675 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:55 smithi073 ceph-mon[32309]: pgmap v345: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:56.363 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.f 2024-03-09T00:17:56.363 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.f 2024-03-09T00:17:56.363 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.f 2024-03-09T00:17:56.364 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.f 2024-03-09T00:17:56.364 DEBUG:tasks.check_counter:Getting stats from i 2024-03-09T00:17:56.364 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.i perf dump 2024-03-09T00:17:56.604 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:17:57.336 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:57 smithi179 ceph-mon[36637]: pgmap v346: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:57.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:57 smithi102 ceph-mon[36106]: pgmap v346: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:57.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:57 smithi073 ceph-mon[32309]: pgmap v346: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-09T00:17:58.335 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-09T00:17:58.335 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:17:58.335 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 1405, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 1391, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 22937622, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 502486, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 6, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 2, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.191391278, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.055157884, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.080395232, 2024-03-09T00:17:58.336 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.050039548, 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 1391, 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.071167256, 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000051162 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.337 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 1606832, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 139008 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 53, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 223, 2024-03-09T00:17:58.338 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 16791346, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 474124, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 7, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 5, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.026071795, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.008165112, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.014219150, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.001174921, 2024-03-09T00:17:58.339 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 220, 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.019448804, 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000088403 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.340 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 457840 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 1346, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 1348, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 13231292, 2024-03-09T00:17:58.341 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 428035, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 5, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 4, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.114027010, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.038786967, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.030424563, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.035576783, 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:17:58.342 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 1346, 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.063199431, 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000046953 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.343 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "cct": { 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "total_workers": 1, 2024-03-09T00:17:58.344 INFO:teuthology.orchestra.run.smithi179.stdout: "unhealthy_workers": 0 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "finisher-MDSRank": { 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "queue_len": 0, 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "complete_latency": { 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 4176, 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.170516338, 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000040832 2024-03-09T00:17:58.345 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "finisher-PurgeQueue": { 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "queue_len": 0, 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "complete_latency": { 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.346 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "mds": { 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "request": 0, 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "reply": 0, 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "reply_latency": { 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.347 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "slow_reply": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "forward": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_fetch_complete": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_fetch_keys": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_commit": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_split": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_merge": 0, 2024-03-09T00:17:58.348 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes": 7877, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_top": 4930, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_bottom": 2947, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_pin_tail": 0, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_pinned": 1745, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_expired": 0, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "inodes_with_caps": 0, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "caps": 0, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "subtrees": 2, 2024-03-09T00:17:58.349 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_hit": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_forward": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_discover": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_dir_fetch": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_remote_ino": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "traverse_lock": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "load_cent": 0, 2024-03-09T00:17:58.350 INFO:teuthology.orchestra.run.smithi179.stdout: "q": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "exported": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "exported_inodes": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "imported": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "imported_inodes": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "openino_dir_fetch": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "openino_backtrace_fetch": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "openino_peer_discover": 0, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "root_rfiles": 3452, 2024-03-09T00:17:58.351 INFO:teuthology.orchestra.run.smithi179.stdout: "root_rbytes": 26509593, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "root_rsnaps": 6, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_backtrace_fetch": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_set_tag": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_backtrace_repaired": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_inotable_repaired": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_dir_inodes": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_dir_base_inodes": 0, 2024-03-09T00:17:58.352 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_dirfrag_rstats": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "scrub_file_inodes": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_inode_file_caps": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_cap_op_revoke": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_cap_op_grant": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_cap_op_trunc": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_client_caps": 0, 2024-03-09T00:17:58.353 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_client_caps_dirty": 0, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_client_cap_release": 0, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "process_request_cap_release": 0 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_cache": { 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_update": 0, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_update_receipt": 0, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_try_discover": 0, 2024-03-09T00:17:58.354 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_send_discover": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "dir_handle_discover": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "num_strays": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "num_strays_delayed": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "num_strays_enqueuing": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "strays_created": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "strays_enqueued": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "strays_reintegrated": 0, 2024-03-09T00:17:58.355 INFO:teuthology.orchestra.run.smithi179.stdout: "strays_migrated": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "num_recovering_processing": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "num_recovering_enqueued": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "num_recovering_prioritized": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "recovery_started": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "recovery_completed": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_enqueue_scrub": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_exportdir": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_flush": 0, 2024-03-09T00:17:58.356 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_fragmentdir": 0, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_fragstats": 0, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "ireq_inodestats": 0 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_log": { 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "evlrg": 0, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "evadd": 0, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "evex": 0, 2024-03-09T00:17:58.357 INFO:teuthology.orchestra.run.smithi179.stdout: "evtrm": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "ev": 23701, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "evexg": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "evexd": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "segadd": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "segex": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "segtrm": 0, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "seg": 6, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "segmjr": 11, 2024-03-09T00:17:58.358 INFO:teuthology.orchestra.run.smithi179.stdout: "segexg": 0, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "segexd": 0, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "expos": 47532541, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "wrpos": 54834278, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "rdpos": 54834142, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "jlat": { 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.359 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "replayed": 23701 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_mem": { 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "ino": 3540, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "ino+": 6755, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "ino-": 3215, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "dir": 248, 2024-03-09T00:17:58.360 INFO:teuthology.orchestra.run.smithi179.stdout: "dir+": 299, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "dir-": 51, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "dn": 7877, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "dn+": 7930, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "dn-": 53, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "cap": 0, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "cap+": 0, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "cap-": 0, 2024-03-09T00:17:58.361 INFO:teuthology.orchestra.run.smithi179.stdout: "rss": 71396, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "heap": 215420 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_server": { 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "dispatch_client_request": 0, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "dispatch_server_request": 0, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_client_request": 0, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_client_session": 0, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "handle_peer_request": 0, 2024-03-09T00:17:58.362 INFO:teuthology.orchestra.run.smithi179.stdout: "req_create_latency": { 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "req_getattr_latency": { 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.363 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: "req_getfilelock_latency": { 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.364 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "req_link_latency": { 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookup_latency": { 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.365 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookuphash_latency": { 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookupino_latency": { 2024-03-09T00:17:58.366 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookupname_latency": { 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.367 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookupparent_latency": { 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lookupsnap_latency": { 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.368 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "req_lssnap_latency": { 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "req_mkdir_latency": { 2024-03-09T00:17:58.369 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.370 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.370 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.370 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "req_mknod_latency": { 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "req_mksnap_latency": { 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.371 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "req_open_latency": { 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.372 INFO:teuthology.orchestra.run.smithi179.stdout: "req_readdir_latency": { 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "req_rename_latency": { 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.373 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "req_renamesnap_latency": { 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "req_snapdiff_latency": { 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.374 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "req_rmdir_latency": { 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "req_rmsnap_latency": { 2024-03-09T00:17:58.375 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "req_rmxattr_latency": { 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.376 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "req_setattr_latency": { 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "req_setdirlayout_latency": { 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.377 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "req_setfilelock_latency": { 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "req_setlayout_latency": { 2024-03-09T00:17:58.378 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "req_setxattr_latency": { 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.379 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "req_symlink_latency": { 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "req_unlink_latency": { 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.380 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "cap_revoke_eviction": 0, 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "cap_acquisition_throttle": 0, 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "req_getvxattr_latency": { 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.381 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_sessions": { 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "session_count": 2, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "session_add": 6, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "session_remove": 4, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "sessions_open": 4, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "sessions_stale": 0, 2024-03-09T00:17:58.382 INFO:teuthology.orchestra.run.smithi179.stdout: "total_load": 0, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "average_load": 0, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "avg_session_uptime": 3438, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "mdthresh_evicted": 0 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "mempool": { 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_items": 0, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:17:58.383 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:17:58.384 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_items": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_items": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:17:58.385 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_items": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:17:58.386 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_items": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_bytes": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_items": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:17:58.387 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_bytes": 337156, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_items": 3597, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_items": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bytes": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_items": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_items": 0, 2024-03-09T00:17:58.388 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_items": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_bytes": 9608, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_items": 184, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_bytes": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_items": 0, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_bytes": 13916483, 2024-03-09T00:17:58.389 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_items": 121485, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_bytes": 0, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_items": 0, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_bytes": 0, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_items": 0 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "objecter": { 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "op_active": 0, 2024-03-09T00:17:58.390 INFO:teuthology.orchestra.run.smithi179.stdout: "op_laggy": 0, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "op_send": 2540, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "op_send_bytes": 26, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "op_resend": 4, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "op_reply": 2540, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "op_latency": { 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 2540, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 2.293952510, 2024-03-09T00:17:58.391 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000903130 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "op_inflight": 0, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "oplen_avg": { 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 2540, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 2541 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "op": 2540, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "op_r": 2540, 2024-03-09T00:17:58.392 INFO:teuthology.orchestra.run.smithi179.stdout: "op_w": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "op_rmw": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "op_pg": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_stat": 1633, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_create": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_read": 904, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_write": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_writefull": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_writesame": 0, 2024-03-09T00:17:58.393 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_append": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_zero": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_truncate": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_delete": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_mapext": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_sparse_read": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_clonerange": 0, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_getxattr": 2, 2024-03-09T00:17:58.394 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_setxattr": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_cmpxattr": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_rmxattr": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_resetxattrs": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_call": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_watch": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_notify": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_src_cmpxattr": 0, 2024-03-09T00:17:58.395 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_pgls": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_pgls_filter": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "osdop_other": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "linger_active": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "linger_send": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "linger_resend": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "linger_ping": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "poolop_active": 0, 2024-03-09T00:17:58.396 INFO:teuthology.orchestra.run.smithi179.stdout: "poolop_send": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "poolop_resend": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "poolstat_active": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "poolstat_send": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "poolstat_resend": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "statfs_active": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "statfs_send": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "statfs_resend": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "command_active": 0, 2024-03-09T00:17:58.397 INFO:teuthology.orchestra.run.smithi179.stdout: "command_send": 0, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "command_resend": 0, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "map_epoch": 104, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "map_full": 0, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "map_inc": 15, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_sessions": 7, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_session_open": 7, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_session_close": 0, 2024-03-09T00:17:58.398 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_laggy": 0, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_wr": 0, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_rd": 2, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_del": 0 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "oft": { 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_total_objs": 0, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_total_kv_pairs": 0, 2024-03-09T00:17:58.399 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_total_updates": 0, 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "omap_total_removes": 0 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "osd-slow-ops": { 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "slow_ops_count": 0 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "purge_queue": { 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executing_ops": 0, 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executing_ops_high_water": 0, 2024-03-09T00:17:58.400 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executing": 0, 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executing_high_water": 0, 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executed_ops": 0, 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_executed": 0, 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "pq_item_in_journal": 0 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:17:58.401 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 104857600, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 2800, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 52692052, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 2800, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:17:58.402 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 2800, 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 52692052, 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.403 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-objecter_bytes": { 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 104857600, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:58.404 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 2540, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 50641332, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 2540, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 50641332, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.405 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-objecter_ops": { 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 1024, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 0, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 0, 2024-03-09T00:17:58.406 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 2540, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 2540, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 2540, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 2540, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.407 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-write_buf_throttle": { 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 3758096384, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 0, 2024-03-09T00:17:58.408 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 0, 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:58.409 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-write_buf_throttle-0x556f63768960": { 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 3758096384, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:17:58.410 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 0, 2024-03-09T00:17:58.411 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:17:58.412 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-09T00:17:59.034 INFO:tasks.check_counter:Daemon mds.i mds.root_rsnaps=6 2024-03-09T00:17:59.035 INFO:tasks.check_counter:Daemon mds.i mds_server.req_mksnap_latency.avgcount=0 2024-03-09T00:17:59.035 INFO:tasks.check_counter:Daemon mds.i mds_server.req_rmsnap_latency.avgcount=0 2024-03-09T00:17:59.035 INFO:tasks.check_counter:Daemon mds.i mds.dir_split=0 2024-03-09T00:17:59.035 DEBUG:tasks.check_counter:Getting stats from l 2024-03-09T00:17:59.035 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761 shell --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 -- ceph daemon mds.l perf dump 2024-03-09T00:17:59.270 INFO:teuthology.orchestra.run.smithi179.stderr:Inferring config /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/mon.c/config 2024-03-09T00:17:59.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:17:59 smithi102 ceph-mon[36106]: pgmap v347: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:59.531 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:17:59 smithi179 ceph-mon[36637]: pgmap v347: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:17:59.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:17:59 smithi073 ceph-mon[32309]: pgmap v347: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:18:01.059 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2024-03-09T00:18:01.060 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-0": { 2024-03-09T00:18:01.060 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 247, 2024-03-09T00:18:01.060 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 234, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 1583488, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 139744, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 4, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 0, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.051190484, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.015711482, 2024-03-09T00:18:01.061 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.037989132, 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.005984592, 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 234, 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.013798876, 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000058969 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:18:01.062 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 1583488, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 139744 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-1": { 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 2, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 177, 2024-03-09T00:18:01.063 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 320, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 186112, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 5, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 3, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.011564151, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.007745670, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.001275243, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:18:01.064 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 174, 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.018853664, 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000108354 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:18:01.065 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 320, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 186112 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "AsyncMessenger::Worker-2": { 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_messages": 0, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages": 2, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_bytes": 0, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_bytes": 0, 2024-03-09T00:18:01.066 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_created_connections": 2, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_active_connections": 1, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_total_time": 0.000368675, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_send_time": 0.000000000, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_recv_time": 0.000300111, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_running_fast_dispatch_time": 0.000000000, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_messages_queue_lat": { 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:18:01.067 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_handle_ack_lat": { 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-09T00:18:01.068 INFO:teuthology.orchestra.run.smithi179.stdout: "msgr_send_encrypted_bytes": 0 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "cct": { 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "total_workers": 0, 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "unhealthy_workers": 0 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "mempool": { 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_bytes": 0, 2024-03-09T00:18:01.069 INFO:teuthology.orchestra.run.smithi179.stdout: "bloom_filter_items": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_bytes": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_alloc_items": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_bytes": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_data_items": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_onode_items": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_meta_items": 0, 2024-03-09T00:18:01.070 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_bytes": 0, 2024-03-09T00:18:01.071 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_other_items": 0, 2024-03-09T00:18:01.071 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-09T00:18:01.071 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_cache_buffer_items": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_bytes": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_extent_items": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_bytes": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_blob_items": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_shared_blob_items": 0, 2024-03-09T00:18:01.072 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_inline_bl_items": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_bytes": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_fsck_items": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_bytes": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_txc_items": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_deferred_items": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_bytes": 0, 2024-03-09T00:18:01.073 INFO:teuthology.orchestra.run.smithi179.stdout: "bluestore_writing_items": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_bytes": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_items": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_bytes": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_reader_items": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_bytes": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "bluefs_file_writer_items": 0, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_bytes": 148744, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_anon_items": 30, 2024-03-09T00:18:01.074 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_bytes": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "buffer_meta_items": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_bytes": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_items": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_bytes": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_mapbl_items": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_bytes": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_pglog_items": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_bytes": 0, 2024-03-09T00:18:01.075 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_items": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_bytes": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "osdmap_mapping_items": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_bytes": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "pgmap_items": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_bytes": 928, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_co_items": 2, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_bytes": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_1_items": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_bytes": 0, 2024-03-09T00:18:01.076 INFO:teuthology.orchestra.run.smithi179.stdout: "unittest_2_items": 0 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "val": 0, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "max": 104857600, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "get_started": 0, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "get": 245, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "get_sum": 1546223, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_fail": 0, 2024-03-09T00:18:01.077 INFO:teuthology.orchestra.run.smithi179.stdout: "get_or_fail_success": 245, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "take": 0, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "take_sum": 0, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "put": 245, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "put_sum": 1546223, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "wait": { 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "avgcount": 0, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "sum": 0.000000000, 2024-03-09T00:18:01.078 INFO:teuthology.orchestra.run.smithi179.stdout: "avgtime": 0.000000000 2024-03-09T00:18:01.079 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:18:01.079 INFO:teuthology.orchestra.run.smithi179.stdout: } 2024-03-09T00:18:01.079 INFO:teuthology.orchestra.run.smithi179.stdout:} 2024-03-09T00:18:01.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:01 smithi102 ceph-mon[36106]: pgmap v348: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:18:01.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:01 smithi179 ceph-mon[36637]: pgmap v348: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:18:01.653 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:01 smithi073 ceph-mon[32309]: pgmap v348: 129 pgs: 129 active+clean; 35 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 853 B/s rd, 1 op/s 2024-03-09T00:18:01.752 WARNING:tasks.check_counter:Counter 'mds.root_rsnaps' not found on daemon mds.l 2024-03-09T00:18:01.752 WARNING:tasks.check_counter:Counter 'mds_server.req_mksnap_latency.avgcount' not found on daemon mds.l 2024-03-09T00:18:01.752 WARNING:tasks.check_counter:Counter 'mds_server.req_rmsnap_latency.avgcount' not found on daemon mds.l 2024-03-09T00:18:01.752 WARNING:tasks.check_counter:Counter 'mds.dir_split' not found on daemon mds.l 2024-03-09T00:18:01.753 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-09T00:18:01.764 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-09T00:18:01.764 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:18:01.764 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-09T00:18:01.782 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-03-09T00:18:01.783 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-09T00:18:01.839 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:18:01.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 -- 172.21.15.73:0/2640921783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f848810dc90 msgr2=0x7f8488110080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:01.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 --2- 172.21.15.73:0/2640921783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f848810dc90 0x7f8488110080 secure :-1 s=READY pgs=1046 cs=0 l=1 rev1=1 crypto rx=0x7f8484009ca0 tx=0x7f848402c0f0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 -- 172.21.15.73:0/2640921783 shutdown_connections 2024-03-09T00:18:01.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 --2- 172.21.15.73:0/2640921783 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f848810dc90 0x7f8488110080 unknown :-1 s=CLOSED pgs=1046 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.986 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 --2- 172.21.15.73:0/2640921783 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f8488100a90 0x7f848810d750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.984+0000 7f848f40b640 1 --2- 172.21.15.73:0/2640921783 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f84881000e0 0x7f84881004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- 172.21.15.73:0/2640921783 >> 172.21.15.73:0/2640921783 conn(0x7f84880772f0 msgr2=0x7f8488077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- 172.21.15.73:0/2640921783 shutdown_connections 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- 172.21.15.73:0/2640921783 wait complete. 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 Processor -- start 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- start start 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84881000e0 0x7f8488199d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:01.987 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f848810dc90 0x7f848819e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84881138b0 con 0x7f8488100a90 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488113730 con 0x7f84881000e0 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848f40b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488113a30 con 0x7f848810dc90 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848c9a6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848c9a6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36642/0 (socket says 172.21.15.73:36642) 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.985+0000 7f848c9a6640 1 -- 172.21.15.73:0/862250749 learned_addr learned my addr 172.21.15.73:0/862250749 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848d9a8640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f848810dc90 0x7f848819e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:01.988 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848d1a7640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84881000e0 0x7f8488199d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 -- 172.21.15.73:0/862250749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f848810dc90 msgr2=0x7f848819e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f848810dc90 0x7f848819e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 -- 172.21.15.73:0/862250749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84881000e0 msgr2=0x7f8488199d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84881000e0 0x7f8488199d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 -- 172.21.15.73:0/862250749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848819ed40 con 0x7f8488100a90 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848c9a6640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f847800c890 tx=0x7f847800cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478004220 con 0x7f8488100a90 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f848819f030 con 0x7f8488100a90 2024-03-09T00:18:01.989 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f84780043c0 con 0x7f8488100a90 2024-03-09T00:18:01.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8478004a00 con 0x7f8488100a90 2024-03-09T00:18:01.990 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.986+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84881a68c0 con 0x7f8488100a90 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.987+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84880680e0 con 0x7f8488100a90 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.988+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f84780079b0 con 0x7f8488100a90 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.989+0000 7f847e7fc640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 0x7f84740a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.989+0000 7f848d1a7640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 0x7f84740a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.989+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f84780ceee0 con 0x7f8488100a90 2024-03-09T00:18:01.991 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.989+0000 7f848d1a7640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 0x7f84740a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8480005e10 tx=0x7f8480005da0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:01.992 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:01.994+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f8478014030 con 0x7f8488100a90 2024-03-09T00:18:02.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.173+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f8488108e50 con 0x7f8488100a90 2024-03-09T00:18:02.173 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.174+0000 7f847e7fc640 1 -- 172.21.15.73:0/862250749 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v104) v1 ==== 74+0+25346 (secure 0 0 0) 0x7f847809b850 con 0x7f8488100a90 2024-03-09T00:18:02.174 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:18:02.174 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":104,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:17:14.975361+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"103","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":17,"snap_epoch":103,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/2159650606":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/102170643":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3270366724":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/2748563172":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[{"pool":3,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]},{"pool":4,"snaps":[{"begin":6,"length":1},{"begin":17,"length":1}]}],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:18:02.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 msgr2=0x7f84740a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 0x7f84740a4e30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f8480005e10 tx=0x7f8480005da0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 msgr2=0x7f848819a290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.176 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 crypto rx=0x7f847800c890 tx=0x7f847800cd50 comp rx=0 tx=0).stop 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 shutdown_connections 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f848810dc90 0x7f848819e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f84740a2970 0x7f84740a4e30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f8488100a90 0x7f848819a290 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 --2- 172.21.15.73:0/862250749 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f84881000e0 0x7f8488199d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 >> 172.21.15.73:0/862250749 conn(0x7f84880772f0 msgr2=0x7f848810f030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 shutdown_connections 2024-03-09T00:18:02.177 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.176+0000 7f848f40b640 1 -- 172.21.15.73:0/862250749 wait complete. 2024-03-09T00:18:02.186 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-08T23:59:30.162428+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-09T00:04:12.258615+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '84', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-09T00:04:12.918227+0000', 'flags': 8193, 'flags_names': 'hashpspool,selfmanaged_snaps', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.05000000074505806}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 4, 'pool_name': 'cephfs_data_ec', 'create_time': '2024-03-09T00:04:16.156967+0000', 'flags': 8197, 'flags_names': 'hashpspool,ec_overwrites,selfmanaged_snaps', 'type': 3, 'size': 4, 'min_size': 3, 'crush_rule': 1, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '103', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 17, 'snap_epoch': 103, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': 'cephfs_data_ec', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 8192, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-09T00:18:02.187 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- 172.21.15.73:0/1650318829 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 msgr2=0x7f351010d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- 172.21.15.73:0/1650318829 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351010d070 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 crypto rx=0x7f3508009ca0 tx=0x7f350802c0f0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- 172.21.15.73:0/1650318829 shutdown_connections 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- 172.21.15.73:0/1650318829 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351010d070 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- 172.21.15.73:0/1650318829 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510100d80 0x7f3510104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- 172.21.15.73:0/1650318829 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 0x7f35101007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- 172.21.15.73:0/1650318829 >> 172.21.15.73:0/1650318829 conn(0x7f35100772f0 msgr2=0x7f3510077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- 172.21.15.73:0/1650318829 shutdown_connections 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- 172.21.15.73:0/1650318829 wait complete. 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 Processor -- start 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- start start 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 0x7f351019d760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.297 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510197910 0x7f3510197dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3510110540 con 0x7f35101054a0 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35101103c0 con 0x7f35101003d0 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.295+0000 7f3518774640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35101106c0 con 0x7f3510197910 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36658/0 (socket says 172.21.15.73:36658) 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 -- 172.21.15.73:0/2147192245 learned_addr learned my addr 172.21.15.73:0/2147192245 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3516d11640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510197910 0x7f3510197dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3516510640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 0x7f351019d760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.298 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 -- 172.21.15.73:0/2147192245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510197910 msgr2=0x7f3510197dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510197910 0x7f3510197dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 -- 172.21.15.73:0/2147192245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 msgr2=0x7f351019d760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 0x7f351019d760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 -- 172.21.15.73:0/2147192245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3510198430 con 0x7f35101054a0 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3515d0f640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3500005fd0 tx=0x7f350000dde0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500017070 con 0x7f35101054a0 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3500009c30 con 0x7f35101054a0 2024-03-09T00:18:02.299 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3510198720 con 0x7f35101054a0 2024-03-09T00:18:02.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3500010460 con 0x7f35101054a0 2024-03-09T00:18:02.300 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.296+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35101a49c0 con 0x7f35101054a0 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.297+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35100680e0 con 0x7f35101054a0 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.298+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f3500003e70 con 0x7f35101054a0 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.299+0000 7f35077fe640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 0x7f34fc0a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.299+0000 7f3516510640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 0x7f34fc0a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.299+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f35000cf630 con 0x7f35101054a0 2024-03-09T00:18:02.301 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.299+0000 7f3516510640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 0x7f34fc0a4eb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3510198a50 tx=0x7f350c005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.302 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.302+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f3500020030 con 0x7f35101054a0 2024-03-09T00:18:02.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.474+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f3510100f40 con 0x7f35101054a0 2024-03-09T00:18:02.474 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.475+0000 7f35077fe640 1 -- 172.21.15.73:0/2147192245 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v104) v1 ==== 93+0+10 (secure 0 0 0) 0x7f350009c020 con 0x7f35101054a0 2024-03-09T00:18:02.474 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 1 2024-03-09T00:18:02.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 msgr2=0x7f34fc0a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.476 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 0x7f34fc0a4eb0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f3510198a50 tx=0x7f350c005f90 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 msgr2=0x7f351019dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 secure :-1 s=READY pgs=1049 cs=0 l=1 rev1=1 crypto rx=0x7f3500005fd0 tx=0x7f350000dde0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 shutdown_connections 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f3510197910 0x7f3510197dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f34fc0a29f0 0x7f34fc0a4eb0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f35101054a0 0x7f351019dca0 unknown :-1 s=CLOSED pgs=1049 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 --2- 172.21.15.73:0/2147192245 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f35101003d0 0x7f351019d760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.477 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 >> 172.21.15.73:0/2147192245 conn(0x7f35100772f0 msgr2=0x7f35101040a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 shutdown_connections 2024-03-09T00:18:02.478 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.477+0000 7f3518774640 1 -- 172.21.15.73:0/2147192245 wait complete. 2024-03-09T00:18:02.485 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:02 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/862250749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:02.487 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-09T00:18:02.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:02 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/862250749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:02.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:02 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/862250749' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:02.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- 172.21.15.73:0/2447719287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541063b0 msgr2=0x7fc054106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 --2- 172.21.15.73:0/2447719287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541063b0 0x7fc054106790 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 crypto rx=0x7fc04c009900 tx=0x7fc04c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:02.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- 172.21.15.73:0/2447719287 shutdown_connections 2024-03-09T00:18:02.594 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 --2- 172.21.15.73:0/2447719287 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc054100e90 0x7fc05410cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 --2- 172.21.15.73:0/2447719287 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc0541004d0 0x7fc054100950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 --2- 172.21.15.73:0/2447719287 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541063b0 0x7fc054106790 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- 172.21.15.73:0/2447719287 >> 172.21.15.73:0/2447719287 conn(0x7fc0540771d0 msgr2=0x7fc0540775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- 172.21.15.73:0/2447719287 shutdown_connections 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- 172.21.15.73:0/2447719287 wait complete. 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 Processor -- start 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 -- start start 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.593+0000 7fc059638640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.595 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc054100e90 0x7fc05419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0541063b0 0x7fc054197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc054110570 con 0x7fc0541004d0 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0541103f0 con 0x7fc054100e90 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0541106f0 con 0x7fc0541063b0 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36668/0 (socket says 172.21.15.73:36668) 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 -- 172.21.15.73:0/942464798 learned_addr learned my addr 172.21.15.73:0/942464798 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:02.596 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc0527fc640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc054100e90 0x7fc05419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc0537fe640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0541063b0 0x7fc054197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 -- 172.21.15.73:0/942464798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0541063b0 msgr2=0x7fc054197910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0541063b0 0x7fc054197910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 -- 172.21.15.73:0/942464798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc054100e90 msgr2=0x7fc05419dc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.597 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc054100e90 0x7fc05419dc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 -- 172.21.15.73:0/942464798 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc054198090 con 0x7fc0541004d0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc052ffd640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc04c006010 tx=0x7fc04c02cc70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c041070 con 0x7fc0541004d0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc04c004450 con 0x7fc0541004d0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc054198320 con 0x7fc0541004d0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc04c034440 con 0x7fc0541004d0 2024-03-09T00:18:02.598 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.594+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0541a45d0 con 0x7fc0541004d0 2024-03-09T00:18:02.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.595+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc054101910 con 0x7fc0541004d0 2024-03-09T00:18:02.599 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.596+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc04c044070 con 0x7fc0541004d0 2024-03-09T00:18:02.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.597+0000 7fc04bfff640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 0x7fc0400a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.597+0000 7fc0527fc640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 0x7fc0400a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.597+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7fc04c0f0d90 con 0x7fc0541004d0 2024-03-09T00:18:02.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.597+0000 7fc0527fc640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 0x7fc0400a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc054105830 tx=0x7fc03c00b410 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.600 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.601+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc04c0bd810 con 0x7fc0541004d0 2024-03-09T00:18:02.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.774+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc0541a47a0 con 0x7fc0541004d0 2024-03-09T00:18:02.774 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.774+0000 7fc04bfff640 1 -- 172.21.15.73:0/942464798 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v104) v1 ==== 104+0+11 (secure 0 0 0) 0x7fc04c0bd110 con 0x7fc0541004d0 2024-03-09T00:18:02.774 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.776+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 msgr2=0x7fc0400a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.776+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 0x7fc0400a4e30 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fc054105830 tx=0x7fc03c00b410 comp rx=0 tx=0).stop 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.776+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 msgr2=0x7fc05419d710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.776+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 crypto rx=0x7fc04c006010 tx=0x7fc04c02cc70 comp rx=0 tx=0).stop 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 shutdown_connections 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc0541063b0 0x7fc054197910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.776 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc054100e90 0x7fc05419dc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc0400a2970 0x7fc0400a4e30 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 --2- 172.21.15.73:0/942464798 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc0541004d0 0x7fc05419d710 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 >> 172.21.15.73:0/942464798 conn(0x7fc0540771d0 msgr2=0x7fc0540ff750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 shutdown_connections 2024-03-09T00:18:02.777 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.777+0000 7fc059638640 1 -- 172.21.15.73:0/942464798 wait complete. 2024-03-09T00:18:02.790 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-09T00:18:02.896 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 -- 172.21.15.73:0/2823171921 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 msgr2=0x7f12ec107720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 --2- 172.21.15.73:0/2823171921 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec107720 secure :-1 s=READY pgs=1052 cs=0 l=1 rev1=1 crypto rx=0x7f12dc009930 tx=0x7f12dc02bc20 comp rx=0 tx=0).stop 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 -- 172.21.15.73:0/2823171921 shutdown_connections 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 --2- 172.21.15.73:0/2823171921 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec06a1e0 0x7f12ec10e170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 --2- 172.21.15.73:0/2823171921 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec069860 0x7f12ec069ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 --2- 172.21.15.73:0/2823171921 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec107720 unknown :-1 s=CLOSED pgs=1052 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 -- 172.21.15.73:0/2823171921 >> 172.21.15.73:0/2823171921 conn(0x7f12ec0771d0 msgr2=0x7f12ec0775e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 -- 172.21.15.73:0/2823171921 shutdown_connections 2024-03-09T00:18:02.897 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 -- 172.21.15.73:0/2823171921 wait complete. 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.895+0000 7f12f1473640 1 Processor -- start 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 -- start start 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec069860 0x7f12ec19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 0x7f12ec19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec1116b0 con 0x7f12ec107340 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec111530 con 0x7f12ec069860 2024-03-09T00:18:02.898 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12f1473640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12ec111830 con 0x7f12ec06a1e0 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36700/0 (socket says 172.21.15.73:36700) 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 -- 172.21.15.73:0/1357285961 learned_addr learned my addr 172.21.15.73:0/1357285961 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eaffd640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec069860 0x7f12ec19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12ea7fc640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 0x7f12ec19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 -- 172.21.15.73:0/1357285961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 msgr2=0x7f12ec19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 0x7f12ec19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.899 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 -- 172.21.15.73:0/1357285961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec069860 msgr2=0x7f12ec19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec069860 0x7f12ec19d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 -- 172.21.15.73:0/1357285961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12ec198150 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12ea7fc640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 0x7f12ec19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.896+0000 7f12eb7fe640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f12e400cce0 tx=0x7f12e400a620 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.897+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e400f450 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.897+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12e400fad0 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.897+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12ec198440 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.897+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12e4018c10 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.897+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12ec1a46e0 con 0x7f12ec107340 2024-03-09T00:18:02.900 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.898+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12ec1029e0 con 0x7f12ec107340 2024-03-09T00:18:02.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.899+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f12e400f5f0 con 0x7f12ec107340 2024-03-09T00:18:02.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.899+0000 7f12d3fff640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 0x7f12d40a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:02.901 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.899+0000 7f12eaffd640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 0x7f12d40a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:02.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.899+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f12e40cf950 con 0x7f12ec107340 2024-03-09T00:18:02.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.900+0000 7f12eaffd640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 0x7f12d40a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12ec198b10 tx=0x7f12dc000bc0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:02.902 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:02.904+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f12e4014030 con 0x7f12ec107340 2024-03-09T00:18:03.075 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.076+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f12ec02ce70 con 0x7f12ec107340 2024-03-09T00:18:03.076 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.076+0000 7f12d3fff640 1 -- 172.21.15.73:0/1357285961 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v104) v1 ==== 100+0+11 (secure 0 0 0) 0x7f12e409c2c0 con 0x7f12ec107340 2024-03-09T00:18:03.076 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 64 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.078+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 msgr2=0x7f12d40a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.078+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 0x7f12d40a4e30 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f12ec198b10 tx=0x7f12dc000bc0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.078+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 msgr2=0x7f12ec1979d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.078+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 crypto rx=0x7f12e400cce0 tx=0x7f12e400a620 comp rx=0 tx=0).stop 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 shutdown_connections 2024-03-09T00:18:03.078 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f12d40a2970 0x7f12d40a4e30 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f12ec107340 0x7f12ec1979d0 unknown :-1 s=CLOSED pgs=1053 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f12ec06a1e0 0x7f12ec19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 --2- 172.21.15.73:0/1357285961 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f12ec069860 0x7f12ec19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 >> 172.21.15.73:0/1357285961 conn(0x7f12ec0771d0 msgr2=0x7f12ec0ff170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 shutdown_connections 2024-03-09T00:18:03.079 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.079+0000 7f12f1473640 1 -- 172.21.15.73:0/1357285961 wait complete. 2024-03-09T00:18:03.088 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data_ec pg_num 2024-03-09T00:18:03.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 -- 172.21.15.73:0/2342931940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941054a0 msgr2=0x7f099410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 --2- 172.21.15.73:0/2342931940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941054a0 0x7f099410d070 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 crypto rx=0x7f098c009ca0 tx=0x7f098c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 -- 172.21.15.73:0/2342931940 shutdown_connections 2024-03-09T00:18:03.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 --2- 172.21.15.73:0/2342931940 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941054a0 0x7f099410d070 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.195 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 --2- 172.21.15.73:0/2342931940 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f0994100d80 0x7f0994104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 --2- 172.21.15.73:0/2342931940 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f09941003d0 0x7f09941007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.193+0000 7f099a95b640 1 -- 172.21.15.73:0/2342931940 >> 172.21.15.73:0/2342931940 conn(0x7f09940772f0 msgr2=0x7f0994077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- 172.21.15.73:0/2342931940 shutdown_connections 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- 172.21.15.73:0/2342931940 wait complete. 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 Processor -- start 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- start start 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09941054a0 0x7f0994198f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.196 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 0x7f09941a3a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0994110560 con 0x7f09941003d0 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941103e0 con 0x7f09941054a0 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.194+0000 7f099a95b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09941106e0 con 0x7f0994199490 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36726/0 (socket says 172.21.15.73:36726) 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 -- 172.21.15.73:0/1774379304 learned_addr learned my addr 172.21.15.73:0/1774379304 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f09937fe640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09941054a0 0x7f0994198f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0998ef8640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 0x7f09941a3a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.197 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 -- 172.21.15.73:0/1774379304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 msgr2=0x7f09941a3a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 0x7f09941a3a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 -- 172.21.15.73:0/1774379304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09941054a0 msgr2=0x7f0994198f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09941054a0 0x7f0994198f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 -- 172.21.15.73:0/1774379304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09941a4210 con 0x7f09941003d0 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0998ef8640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 0x7f09941a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f0993fff640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f098400d770 tx=0x7f098400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984017070 con 0x7f09941003d0 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09941a4500 con 0x7f09941003d0 2024-03-09T00:18:03.198 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0984005500 con 0x7f09941003d0 2024-03-09T00:18:03.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0984004e00 con 0x7f09941003d0 2024-03-09T00:18:03.199 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.195+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09941a4a50 con 0x7f09941003d0 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.196+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09940680e0 con 0x7f09941003d0 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.197+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f0984004fa0 con 0x7f09941003d0 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.198+0000 7f09917fa640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 0x7f097c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.198+0000 7f09937fe640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 0x7f097c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.198+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f09840cf0e0 con 0x7f09941003d0 2024-03-09T00:18:03.200 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.198+0000 7f09937fe640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 0x7f097c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0994068fa0 tx=0x7f0988005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:03.201 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.201+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f098409ba50 con 0x7f09941003d0 2024-03-09T00:18:03.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.375+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"} v 0) v1 -- 0x7f0994108730 con 0x7f09941003d0 2024-03-09T00:18:03.374 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.375+0000 7f09917fa640 1 -- 172.21.15.73:0/1774379304 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]=0 v104) v1 ==== 103+0+11 (secure 0 0 0) 0x7f098409b350 con 0x7f09941003d0 2024-03-09T00:18:03.375 INFO:teuthology.orchestra.run.smithi073.stdout:pg_num: 32 2024-03-09T00:18:03.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 msgr2=0x7f097c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 0x7f097c0a4e30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0994068fa0 tx=0x7f0988005f90 comp rx=0 tx=0).stop 2024-03-09T00:18:03.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 msgr2=0x7f09941989c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 crypto rx=0x7f098400d770 tx=0x7f098400dc30 comp rx=0 tx=0).stop 2024-03-09T00:18:03.377 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 shutdown_connections 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f0994199490 0x7f09941a3a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f09941054a0 0x7f0994198f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f097c0a2970 0x7f097c0a4e30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 --2- 172.21.15.73:0/1774379304 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f09941003d0 0x7f09941989c0 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 >> 172.21.15.73:0/1774379304 conn(0x7f09940772f0 msgr2=0x7f09940fe230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 shutdown_connections 2024-03-09T00:18:03.378 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.377+0000 7f099a95b640 1 -- 172.21.15.73:0/1774379304 wait complete. 2024-03-09T00:18:03.386 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:03 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2147192245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.386 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:03 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/942464798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.386 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:03 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1357285961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.386 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:03 smithi073 ceph-mon[32309]: pgmap v349: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:18:03.387 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-09T00:18:03.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/1922946536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb00ffd50 msgr2=0x7ffbb0100130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/1922946536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb00ffd50 0x7ffbb0100130 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 crypto rx=0x7ffba4009900 tx=0x7ffba402bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:03.492 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/1922946536 shutdown_connections 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/1922946536 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb010e100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/1922946536 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb0100700 0x7ffbb0105e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/1922946536 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb00ffd50 0x7ffbb0100130 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/1922946536 >> 172.21.15.73:0/1922946536 conn(0x7ffbb00753d0 msgr2=0x7ffbb00757e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/1922946536 shutdown_connections 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/1922946536 wait complete. 2024-03-09T00:18:03.493 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.490+0000 7ffbb5ba2640 1 Processor -- start 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 -- start start 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb00ffd50 0x7ffbb019d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb019dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb0111640 con 0x7ffbb0197920 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb01114c0 con 0x7ffbb00ffd50 2024-03-09T00:18:03.494 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbb01117c0 con 0x7ffbb0106510 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36746/0 (socket says 172.21.15.73:36746) 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 -- 172.21.15.73:0/686397089 learned_addr learned my addr 172.21.15.73:0/686397089 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbaffff640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb019dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb4ba0640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb00ffd50 0x7ffbb019d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 -- 172.21.15.73:0/686397089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 msgr2=0x7ffbb019dcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb019dcb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.495 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 -- 172.21.15.73:0/686397089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb00ffd50 msgr2=0x7ffbb019d770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb00ffd50 0x7ffbb019d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 -- 172.21.15.73:0/686397089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbb0198440 con 0x7ffbb0197920 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbaffff640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb53a1640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ffba800b970 tx=0x7ffba800be30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8019070 con 0x7ffbb0197920 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffba800cb40 con 0x7ffbb0197920 2024-03-09T00:18:03.496 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffba8007650 con 0x7ffbb0197920 2024-03-09T00:18:03.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.491+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbb0198730 con 0x7ffbb0197920 2024-03-09T00:18:03.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.492+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbb01a4930 con 0x7ffbb0197920 2024-03-09T00:18:03.497 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.493+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbb0068450 con 0x7ffbb0197920 2024-03-09T00:18:03.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.493+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ffba801c070 con 0x7ffbb0197920 2024-03-09T00:18:03.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.494+0000 7ffbadffb640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 0x7ffba00a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:03.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.494+0000 7ffbb4ba0640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 0x7ffba00a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:03.498 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.494+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7ffba80ce830 con 0x7ffbb0197920 2024-03-09T00:18:03.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.495+0000 7ffbb4ba0640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 0x7ffba00a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffba4036040 tx=0x7ffba4005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:03.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.499+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ffba809b250 con 0x7ffbb0197920 2024-03-09T00:18:03.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:03 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2147192245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:03 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/942464798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:03 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1357285961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:03 smithi102 ceph-mon[36106]: pgmap v349: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:18:03.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:03 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2147192245' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:03 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/942464798' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:03 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1357285961' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-09T00:18:03.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:03 smithi179 ceph-mon[36637]: pgmap v349: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 44 KiB/s wr, 19 op/s 2024-03-09T00:18:03.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.686+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7ffbb0198e50 con 0x7ffbb0197920 2024-03-09T00:18:03.685 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.686+0000 7ffbadffb640 1 -- 172.21.15.73:0/686397089 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 16 entries v104) v1 ==== 81+0+942 (secure 0 0 0) 0x7ffba809ab50 con 0x7ffbb0197920 2024-03-09T00:18:03.686 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/2159650606 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.686 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/2701889057 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.686 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/102170643 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.686 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/3270366724 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.686 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/2701889057 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/4041937587 2024-03-09T23:57:03.831065+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/2748563172 2024-03-10T00:06:50.942952+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/3726094398 2024-03-09T23:57:03.831065+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6800/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/1410011990 2024-03-09T23:56:36.296712+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/545419019 2024-03-09T23:57:03.831065+0000 2024-03-09T00:18:03.687 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:6801/1574449933 2024-03-09T23:56:36.296712+0000 2024-03-09T00:18:03.688 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/4143923877 2024-03-09T23:56:36.296712+0000 2024-03-09T00:18:03.688 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/161900832 2024-03-09T23:57:03.831065+0000 2024-03-09T00:18:03.688 INFO:teuthology.orchestra.run.smithi073.stdout:172.21.15.73:0/383471347 2024-03-09T23:56:36.296712+0000 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:listed 16 entries 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 msgr2=0x7ffba00a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 0x7ffba00a4e30 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7ffba4036040 tx=0x7ffba4005fd0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 msgr2=0x7ffbb0197dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 secure :-1 s=READY pgs=1057 cs=0 l=1 rev1=1 crypto rx=0x7ffba800b970 tx=0x7ffba800be30 comp rx=0 tx=0).stop 2024-03-09T00:18:03.689 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 shutdown_connections 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ffba00a2970 0x7ffba00a4e30 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ffbb0197920 0x7ffbb0197dd0 unknown :-1 s=CLOSED pgs=1057 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ffbb0106510 0x7ffbb019dcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 --2- 172.21.15.73:0/686397089 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ffbb00ffd50 0x7ffbb019d770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 >> 172.21.15.73:0/686397089 conn(0x7ffbb00753d0 msgr2=0x7ffbb00fdd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 shutdown_connections 2024-03-09T00:18:03.690 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:03.688+0000 7ffbb5ba2640 1 -- 172.21.15.73:0/686397089 wait complete. 2024-03-09T00:18:03.698 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-09T00:18:03.698 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:18:03.698 DEBUG:teuthology.orchestra.run.smithi073:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-09T00:18:03.802 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:18:03.802 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:18:03.803 DEBUG:teuthology.orchestra.run.smithi073:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-09T00:18:03.818 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:18:03.818 DEBUG:teuthology.orchestra.run.smithi073:> ip netns list 2024-03-09T00:18:03.873 INFO:teuthology.orchestra.run.smithi073.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-09T00:18:03.874 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:18:03.874 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-09T00:18:03.941 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-09T00:18:03.941 DEBUG:teuthology.orchestra.run.smithi073:> sudo ip link delete ceph-brx 2024-03-09T00:18:04.029 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-09T00:18:04.041 INFO:tasks.cephadm:Teardown begin 2024-03-09T00:18:04.041 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-09T00:18:04.041 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:18:04.144 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 -- 172.21.15.73:0/4082449583 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 msgr2=0x7f26a0102960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 --2- 172.21.15.73:0/4082449583 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a0102960 secure :-1 s=READY pgs=1058 cs=0 l=1 rev1=1 crypto rx=0x7f269c009900 tx=0x7f269c02bcd0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 -- 172.21.15.73:0/4082449583 shutdown_connections 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 --2- 172.21.15.73:0/4082449583 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a010dda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 --2- 172.21.15.73:0/4082449583 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 0x7f26a010b380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 --2- 172.21.15.73:0/4082449583 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a0102960 unknown :-1 s=CLOSED pgs=1058 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 -- 172.21.15.73:0/4082449583 >> 172.21.15.73:0/4082449583 conn(0x7f26a0077350 msgr2=0x7f26a0077760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 -- 172.21.15.73:0/4082449583 shutdown_connections 2024-03-09T00:18:04.145 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.145+0000 7f26a733b640 1 -- 172.21.15.73:0/4082449583 wait complete. 2024-03-09T00:18:04.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 Processor -- start 2024-03-09T00:18:04.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 -- start start 2024-03-09T00:18:04.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.148 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 0x7f26a019de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a0197ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a0111750 con 0x7f26a0102580 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01115d0 con 0x7f26a010b950 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a733b640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26a01118d0 con 0x7f26a0102ea0 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36774/0 (socket says 172.21.15.73:36774) 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 -- 172.21.15.73:0/2834224608 learned_addr learned my addr 172.21.15.73:0/2834224608 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:04.149 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a58d8640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a0197ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 -- 172.21.15.73:0/2834224608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 msgr2=0x7f26a019de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a48d6640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 0x7f26a019de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 0x7f26a019de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 -- 172.21.15.73:0/2834224608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 msgr2=0x7f26a0197ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a0197ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 -- 172.21.15.73:0/2834224608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26a0198230 con 0x7f26a0102580 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a58d8640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.146+0000 7f26a50d7640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f269c005940 tx=0x7f269c02cd10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.150 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.147+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c041070 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.147+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f269c004450 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.147+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f269c034440 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.147+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26a01984c0 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.147+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26a01a47c0 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.148+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26a0068530 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.149+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f269c044070 con 0x7f26a0102580 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.149+0000 7f26967fc640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 0x7f268c0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.151 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.149+0000 7f26a48d6640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 0x7f268c0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.149+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9525+0+0 (secure 0 0 0) 0x7f269c0f1040 con 0x7f26a0102580 2024-03-09T00:18:04.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.150+0000 7f26a48d6640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 0x7f268c0a4e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26a00ffc20 tx=0x7f2690008040 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.152 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.153+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f269c0bdac0 con 0x7f26a0102580 2024-03-09T00:18:04.380 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.381+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f26a0106a40 con 0x7f26a0102580 2024-03-09T00:18:04.381 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.381+0000 7f26967fc640 1 -- 172.21.15.73:0/2834224608 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7f269c0bd3c0 con 0x7f26a0102580 2024-03-09T00:18:04.381 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:18:04.381 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:18:04.382 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:18:04.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 msgr2=0x7f268c0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 0x7f268c0a4e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f26a00ffc20 tx=0x7f2690008040 comp rx=0 tx=0).stop 2024-03-09T00:18:04.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 msgr2=0x7f26a019d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.383 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 crypto rx=0x7f269c005940 tx=0x7f269c02cd10 comp rx=0 tx=0).stop 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 shutdown_connections 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f26a010b950 0x7f26a0197ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f26a0102ea0 0x7f26a019de40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f268c0a2970 0x7f268c0a4e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 --2- 172.21.15.73:0/2834224608 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f26a0102580 0x7f26a019d900 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 >> 172.21.15.73:0/2834224608 conn(0x7f26a0077350 msgr2=0x7f26a00fd320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:04.384 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 shutdown_connections 2024-03-09T00:18:04.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.384+0000 7f26a733b640 1 -- 172.21.15.73:0/2834224608 wait complete. 2024-03-09T00:18:04.394 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-09T00:18:04.499 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.497+0000 7effe6044640 1 -- 172.21.15.73:0/177643589 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 msgr2=0x7effe010b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.497+0000 7effe6044640 1 --2- 172.21.15.73:0/177643589 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe010b330 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 crypto rx=0x7effd4009900 tx=0x7effd402bcd0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- 172.21.15.73:0/177643589 shutdown_connections 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- 172.21.15.73:0/177643589 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe010dd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- 172.21.15.73:0/177643589 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe010b330 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- 172.21.15.73:0/177643589 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 0x7effe0102910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.500 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- 172.21.15.73:0/177643589 >> 172.21.15.73:0/177643589 conn(0x7effe0076570 msgr2=0x7effe0076980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- 172.21.15.73:0/177643589 shutdown_connections 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- 172.21.15.73:0/177643589 wait complete. 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 Processor -- start 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- start start 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 0x7effe019d880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe0197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.501 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0111690 con 0x7effe0102e50 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0111510 con 0x7effe010b900 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.498+0000 7effe6044640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effe0111810 con 0x7effe0102530 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36796/0 (socket says 172.21.15.73:36796) 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 -- 172.21.15.73:0/1698694821 learned_addr learned my addr 172.21.15.73:0/1698694821 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdffff640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe0197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdf7fe640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 0x7effe019d880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 -- 172.21.15.73:0/1698694821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 msgr2=0x7effe019d880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.502 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 0x7effe019d880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 -- 172.21.15.73:0/1698694821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 msgr2=0x7effe0197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe0197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 -- 172.21.15.73:0/1698694821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effe01981b0 con 0x7effe0102e50 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdffff640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdeffd640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7effd4005940 tx=0x7effd402cbd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4041070 con 0x7effe0102e50 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effe0198440 con 0x7effe0102e50 2024-03-09T00:18:04.503 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7effd4004450 con 0x7effe0102e50 2024-03-09T00:18:04.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7effd4003d80 con 0x7effe0102e50 2024-03-09T00:18:04.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.499+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effe01a4740 con 0x7effe0102e50 2024-03-09T00:18:04.504 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.500+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effe006a160 con 0x7effe0102e50 2024-03-09T00:18:04.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.501+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7effd4044070 con 0x7effe0102e50 2024-03-09T00:18:04.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.502+0000 7effdcff9640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 0x7effc80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.502+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9613+0+0 (secure 0 0 0) 0x7effd40f0dd0 con 0x7effe0102e50 2024-03-09T00:18:04.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.502+0000 7effdf7fe640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 0x7effc80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.505 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.502+0000 7effdf7fe640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 0x7effc80a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7effe0198b70 tx=0x7effd0005f90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.506 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.506+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7effd40bd800 con 0x7effe0102e50 2024-03-09T00:18:04.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1774379304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:18:04.513 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:04 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/686397089' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:18:04.585 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1774379304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:18:04.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:04 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/686397089' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:18:04.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1774379304' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data_ec", "var": "pg_num"}]: dispatch 2024-03-09T00:18:04.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:04 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/686397089' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-09T00:18:04.739 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.740+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7effe01069f0 con 0x7effe0102e50 2024-03-09T00:18:04.740 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.741+0000 7effdcff9640 1 -- 172.21.15.73:0/1698694821 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+11301 (secure 0 0 0) 0x7effd40bd100 con 0x7effe0102e50 2024-03-09T00:18:04.741 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:18:04.741 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":20,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14556,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6835/1664042644","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6834","nonce":1664042644},{"type":"v1","addr":"172.21.15.73:6835","nonce":1664042644}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14568,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6837/3028984722","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6836","nonce":3028984722},{"type":"v1","addr":"172.21.15.73:6837","nonce":3028984722}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14631,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6839/2046390468","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6838","nonce":2046390468},{"type":"v1","addr":"172.21.15.73:6839","nonce":2046390468}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":14649,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.73:6841/2662738077","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6840","nonce":2662738077},{"type":"v1","addr":"172.21.15.73:6841","nonce":2662738077}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24406,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6833/3042063350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6832","nonce":3042063350},{"type":"v1","addr":"172.21.15.179:6833","nonce":3042063350}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24410,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6833/2292586216","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6832","nonce":2292586216},{"type":"v1","addr":"172.21.15.102:6833","nonce":2292586216}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24446,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6835/364340358","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6834","nonce":364340358},{"type":"v1","addr":"172.21.15.102:6835","nonce":364340358}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24457,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6835/2774016402","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6834","nonce":2774016402},{"type":"v1","addr":"172.21.15.179:6835","nonce":2774016402}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24475,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.179:6837/2684774857","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6836","nonce":2684774857},{"type":"v1","addr":"172.21.15.179:6837","nonce":2684774857}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24479,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.102:6837/306199757","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6836","nonce":306199757},{"type":"v1","addr":"172.21.15.102:6837","nonce":306199757}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[{"mdsmap":{"epoch":20,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-09T00:04:13.661086+0000","modified":"2024-03-09T00:04:39.321347+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":1,"in":[0],"up":{"mds_0":24515},"failed":[],"damaged":[],"stopped":[],"info":{"gid_24493":{"gid":24493,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.179:6839/2754171892","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6838","nonce":2754171892},{"type":"v1","addr":"172.21.15.179:6839","nonce":2754171892}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.102:6839/3264474111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6838","nonce":3264474111},{"type":"v1","addr":"172.21.15.102:6839","nonce":3264474111}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3,4],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0},"id":1}]} 2024-03-09T00:18:04.741 INFO:teuthology.orchestra.run.smithi073.stderr:dumped fsmap epoch 20 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 msgr2=0x7effc80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 0x7effc80a4e30 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7effe0198b70 tx=0x7effd0005f90 comp rx=0 tx=0).stop 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 msgr2=0x7effe019ddc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 secure :-1 s=READY pgs=1061 cs=0 l=1 rev1=1 crypto rx=0x7effd4005940 tx=0x7effd402cbd0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 shutdown_connections 2024-03-09T00:18:04.743 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7effe010b900 0x7effe0197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7effc80a2970 0x7effc80a4e30 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7effe0102e50 0x7effe019ddc0 unknown :-1 s=CLOSED pgs=1061 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 --2- 172.21.15.73:0/1698694821 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7effe0102530 0x7effe019d880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 >> 172.21.15.73:0/1698694821 conn(0x7effe0076570 msgr2=0x7effe00fe220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 shutdown_connections 2024-03-09T00:18:04.744 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.743+0000 7effe6044640 1 -- 172.21.15.73:0/1698694821 wait complete. 2024-03-09T00:18:04.754 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-09T00:18:04.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 -- 172.21.15.73:0/3043483627 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef0100a90 msgr2=0x7fdef010d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.858 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 --2- 172.21.15.73:0/3043483627 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef0100a90 0x7fdef010d750 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 crypto rx=0x7fdee0009900 tx=0x7fdee002bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 -- 172.21.15.73:0/3043483627 shutdown_connections 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 --2- 172.21.15.73:0/3043483627 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef010dc90 0x7fdef0110080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 --2- 172.21.15.73:0/3043483627 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef0100a90 0x7fdef010d750 unknown :-1 s=CLOSED pgs=1062 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 --2- 172.21.15.73:0/3043483627 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef01000e0 0x7fdef01004c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 -- 172.21.15.73:0/3043483627 >> 172.21.15.73:0/3043483627 conn(0x7fdef00772f0 msgr2=0x7fdef0077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 -- 172.21.15.73:0/3043483627 shutdown_connections 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.856+0000 7fdef7b59640 1 -- 172.21.15.73:0/3043483627 wait complete. 2024-03-09T00:18:04.859 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 Processor -- start 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 -- start start 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef0100a90 0x7fdef019a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 0x7fdef019e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef01138c0 con 0x7fdef01000e0 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0113740 con 0x7fdef010dc90 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef7b59640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdef0113a40 con 0x7fdef0100a90 2024-03-09T00:18:04.860 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef58f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef58f5640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36820/0 (socket says 172.21.15.73:36820) 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef58f5640 1 -- 172.21.15.73:0/612904883 learned_addr learned my addr 172.21.15.73:0/612904883 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef60f6640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 0x7fdef019e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.857+0000 7fdef50f4640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef0100a90 0x7fdef019a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 -- 172.21.15.73:0/612904883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef0100a90 msgr2=0x7fdef019a2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef0100a90 0x7fdef019a2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 -- 172.21.15.73:0/612904883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 msgr2=0x7fdef019e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 0x7fdef019e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:04.861 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 -- 172.21.15.73:0/612904883 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdef019ed90 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef60f6640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 0x7fdef019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef58f5640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c890 tx=0x7fdeec00cd50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec004220 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdeec0043c0 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdef019f080 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdeec0049e0 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.858+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdef01a6910 con 0x7fdef01000e0 2024-03-09T00:18:04.862 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.859+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdef00680e0 con 0x7fdef01000e0 2024-03-09T00:18:04.863 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.860+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fdeec0079b0 con 0x7fdef01000e0 2024-03-09T00:18:04.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.861+0000 7fdee6ffd640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 0x7fdedc0a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:04.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.861+0000 7fdef50f4640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 0x7fdedc0a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:04.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.861+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9613+0+0 (secure 0 0 0) 0x7fdeec0cf3c0 con 0x7fdef01000e0 2024-03-09T00:18:04.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.861+0000 7fdef50f4640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 0x7fdedc0a4e30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdef00ffbc0 tx=0x7fdee002cf90 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:04.864 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:04.864+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fdeec014030 con 0x7fdef01000e0 2024-03-09T00:18:05.046 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.047+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdef019b0f0 con 0x7fdef01000e0 2024-03-09T00:18:05.047 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.048+0000 7fdee6ffd640 1 -- 172.21.15.73:0/612904883 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v105) v1 ==== 74+0+25481 (secure 0 0 0) 0x7fdeec09bce0 con 0x7fdef01000e0 2024-03-09T00:18:05.048 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-03-09T00:18:05.048 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":105,"fsid":"44f886a8-dda7-11ee-95c6-87774f69a715","created":"2024-03-08T23:56:09.713229+0000","modified":"2024-03-09T00:18:04.261723+0000","last_up_change":"2024-03-09T00:03:21.185524+0000","last_in_change":"2024-03-09T00:02:59.990056+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":27,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-08T23:59:30.162428+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-09T00:04:12.258615+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"84","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-09T00:04:12.918227+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.05000000074505806},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":4,"pool_name":"cephfs_data_ec","create_time":"2024-03-09T00:04:16.156967+0000","flags":8197,"flags_names":"hashpspool,ec_overwrites,selfmanaged_snaps","type":3,"size":4,"min_size":3,"crush_rule":1,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"105","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":19,"snap_epoch":105,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"cephfs_data_ec","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":8192,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"23a8d6f6-f2b8-449c-8948-2a7287b837a9","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6803","nonce":3305091634}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6805","nonce":3305091634}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6809","nonce":3305091634}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":3305091634},{"type":"v1","addr":"172.21.15.73:6807","nonce":3305091634}]},"public_addr":"172.21.15.73:6803/3305091634","cluster_addr":"172.21.15.73:6805/3305091634","heartbeat_back_addr":"172.21.15.73:6809/3305091634","heartbeat_front_addr":"172.21.15.73:6807/3305091634","state":["exists","up"]},{"osd":1,"uuid":"c4e798b8-2a59-4ab9-8e1b-687cb0a3099d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6800","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6801","nonce":491071057}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6802","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6803","nonce":491071057}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6806","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6807","nonce":491071057}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6804","nonce":491071057},{"type":"v1","addr":"172.21.15.102:6805","nonce":491071057}]},"public_addr":"172.21.15.102:6801/491071057","cluster_addr":"172.21.15.102:6803/491071057","heartbeat_back_addr":"172.21.15.102:6807/491071057","heartbeat_front_addr":"172.21.15.102:6805/491071057","state":["exists","up"]},{"osd":2,"uuid":"ba842f27-768e-4c79-827f-4e0ca1411d87","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6800","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6801","nonce":2018456550}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6802","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6803","nonce":2018456550}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6806","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6807","nonce":2018456550}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6804","nonce":2018456550},{"type":"v1","addr":"172.21.15.179:6805","nonce":2018456550}]},"public_addr":"172.21.15.179:6801/2018456550","cluster_addr":"172.21.15.179:6803/2018456550","heartbeat_back_addr":"172.21.15.179:6807/2018456550","heartbeat_front_addr":"172.21.15.179:6805/2018456550","state":["exists","up"]},{"osd":3,"uuid":"b9390f11-f076-4591-b204-73187536054b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":25,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6811","nonce":3007966557}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6813","nonce":3007966557}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6817","nonce":3007966557}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":3007966557},{"type":"v1","addr":"172.21.15.73:6815","nonce":3007966557}]},"public_addr":"172.21.15.73:6811/3007966557","cluster_addr":"172.21.15.73:6813/3007966557","heartbeat_back_addr":"172.21.15.73:6817/3007966557","heartbeat_front_addr":"172.21.15.73:6815/3007966557","state":["exists","up"]},{"osd":4,"uuid":"be6f4676-0213-47ac-806e-52fe6bddb9e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":33,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6808","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6809","nonce":1213471038}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6810","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6811","nonce":1213471038}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6814","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6815","nonce":1213471038}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6812","nonce":1213471038},{"type":"v1","addr":"172.21.15.102:6813","nonce":1213471038}]},"public_addr":"172.21.15.102:6809/1213471038","cluster_addr":"172.21.15.102:6811/1213471038","heartbeat_back_addr":"172.21.15.102:6815/1213471038","heartbeat_front_addr":"172.21.15.102:6813/1213471038","state":["exists","up"]},{"osd":5,"uuid":"8b052756-28e2-438c-ad07-0e489a79f2f2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":38,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6808","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6809","nonce":833646001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6810","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6811","nonce":833646001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6814","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6815","nonce":833646001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6812","nonce":833646001},{"type":"v1","addr":"172.21.15.179:6813","nonce":833646001}]},"public_addr":"172.21.15.179:6809/833646001","cluster_addr":"172.21.15.179:6811/833646001","heartbeat_back_addr":"172.21.15.179:6815/833646001","heartbeat_front_addr":"172.21.15.179:6813/833646001","state":["exists","up"]},{"osd":6,"uuid":"4be6204e-f4a1-4605-a89f-3b3e222336c1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":46,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6819","nonce":1949163006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6821","nonce":1949163006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6825","nonce":1949163006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":1949163006},{"type":"v1","addr":"172.21.15.73:6823","nonce":1949163006}]},"public_addr":"172.21.15.73:6819/1949163006","cluster_addr":"172.21.15.73:6821/1949163006","heartbeat_back_addr":"172.21.15.73:6825/1949163006","heartbeat_front_addr":"172.21.15.73:6823/1949163006","state":["exists","up"]},{"osd":7,"uuid":"e08563fd-4504-49ad-9b28-33db12548e5e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":51,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6816","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6817","nonce":1887369896}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6818","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6819","nonce":1887369896}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6822","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6823","nonce":1887369896}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6820","nonce":1887369896},{"type":"v1","addr":"172.21.15.102:6821","nonce":1887369896}]},"public_addr":"172.21.15.102:6817/1887369896","cluster_addr":"172.21.15.102:6819/1887369896","heartbeat_back_addr":"172.21.15.102:6823/1887369896","heartbeat_front_addr":"172.21.15.102:6821/1887369896","state":["exists","up"]},{"osd":8,"uuid":"0b60fba7-28c7-455e-9120-944eaa7f08dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":59,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6816","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6817","nonce":224478801}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6818","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6819","nonce":224478801}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6822","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6823","nonce":224478801}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6820","nonce":224478801},{"type":"v1","addr":"172.21.15.179:6821","nonce":224478801}]},"public_addr":"172.21.15.179:6817/224478801","cluster_addr":"172.21.15.179:6819/224478801","heartbeat_back_addr":"172.21.15.179:6823/224478801","heartbeat_front_addr":"172.21.15.179:6821/224478801","state":["exists","up"]},{"osd":9,"uuid":"f6075430-0ebe-4194-9acd-b33cbed4bdd1","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":67,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6827","nonce":1773882248}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6828","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6829","nonce":1773882248}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6832","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6833","nonce":1773882248}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6830","nonce":1773882248},{"type":"v1","addr":"172.21.15.73:6831","nonce":1773882248}]},"public_addr":"172.21.15.73:6827/1773882248","cluster_addr":"172.21.15.73:6829/1773882248","heartbeat_back_addr":"172.21.15.73:6833/1773882248","heartbeat_front_addr":"172.21.15.73:6831/1773882248","state":["exists","up"]},{"osd":10,"uuid":"9efc1e41-835a-496e-8487-b158cca2925a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":72,"up_thru":90,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6824","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6825","nonce":2526045067}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6826","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6827","nonce":2526045067}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6830","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6831","nonce":2526045067}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.102:6828","nonce":2526045067},{"type":"v1","addr":"172.21.15.102:6829","nonce":2526045067}]},"public_addr":"172.21.15.102:6825/2526045067","cluster_addr":"172.21.15.102:6827/2526045067","heartbeat_back_addr":"172.21.15.102:6831/2526045067","heartbeat_front_addr":"172.21.15.102:6829/2526045067","state":["exists","up"]},{"osd":11,"uuid":"48124e49-5bc5-49ff-b7b4-c139fcb1346f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":80,"up_thru":92,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6824","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6825","nonce":897506355}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6826","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6827","nonce":897506355}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6830","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6831","nonce":897506355}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.179:6828","nonce":897506355},{"type":"v1","addr":"172.21.15.179:6829","nonce":897506355}]},"public_addr":"172.21.15.179:6825/897506355","cluster_addr":"172.21.15.179:6827/897506355","heartbeat_back_addr":"172.21.15.179:6831/897506355","heartbeat_front_addr":"172.21.15.179:6829/897506355","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:58:40.666949+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:04.481559+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:27.637475+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-08T23:59:53.114147+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:17.975300+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:00:43.130129+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:08.880553+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:34.385400+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:01:59.491789+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:26.084153+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:02:51.845234+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-09T00:03:18.488976+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.5","mappings":[{"from":8,"to":11}]},{"pgid":"2.10","mappings":[{"from":8,"to":11}]},{"pgid":"2.16","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":2,"to":4}]},{"pgid":"3.2","mappings":[{"from":3,"to":9}]},{"pgid":"3.e","mappings":[{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.1b","mappings":[{"from":4,"to":6}]},{"pgid":"3.3a","mappings":[{"from":4,"to":8}]},{"pgid":"4.7","mappings":[{"from":5,"to":8}]},{"pgid":"4.10","mappings":[{"from":9,"to":8}]},{"pgid":"4.12","mappings":[{"from":9,"to":10}]},{"pgid":"4.19","mappings":[{"from":9,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.73:0/2159650606":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/102170643":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3270366724":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6801/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/2701889057":"2024-03-10T00:06:50.942952+0000","172.21.15.73:6800/4041937587":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/2748563172":"2024-03-10T00:06:50.942952+0000","172.21.15.73:0/3726094398":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6800/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/1410011990":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/545419019":"2024-03-09T23:57:03.831065+0000","172.21.15.73:6801/1574449933":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/4143923877":"2024-03-09T23:56:36.296712+0000","172.21.15.73:0/161900832":"2024-03-09T23:57:03.831065+0000","172.21.15.73:0/383471347":"2024-03-09T23:56:36.296712+0000"},"range_blocklist":{},"erasure_code_profiles":{"cephfs_data_ec":{"crush-device-class":"","crush-failure-domain":"osd","crush-num-failure-domains":"0","crush-osds-per-failure-domain":"0","crush-root":"default","jerasure-per-chunk-alignment":"false","k":"2","m":"2","plugin":"jerasure","technique":"reed_sol_van","w":"8"},"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_removed_snaps":[{"pool":3,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]},{"pool":4,"snaps":[{"begin":7,"length":1},{"begin":19,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-09T00:18:05.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.050+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 msgr2=0x7fdedc0a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.050+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 0x7fdedc0a4e30 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7fdef00ffbc0 tx=0x7fdee002cf90 comp rx=0 tx=0).stop 2024-03-09T00:18:05.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.050+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 msgr2=0x7fdef0199da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.050+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 crypto rx=0x7fdeec00c890 tx=0x7fdeec00cd50 comp rx=0 tx=0).stop 2024-03-09T00:18:05.050 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.050+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 shutdown_connections 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fdef010dc90 0x7fdef019e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fdef0100a90 0x7fdef019a2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fdedc0a2970 0x7fdedc0a4e30 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 --2- 172.21.15.73:0/612904883 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fdef01000e0 0x7fdef0199da0 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 >> 172.21.15.73:0/612904883 conn(0x7fdef00772f0 msgr2=0x7fdef010f110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 shutdown_connections 2024-03-09T00:18:05.051 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.051+0000 7fdef7b59640 1 -- 172.21.15.73:0/612904883 wait complete. 2024-03-09T00:18:05.061 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-09T00:18:05.164 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 -- 172.21.15.73:0/2072885092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f01065c0 msgr2=0x7ff8f010e190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2072885092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f01065c0 0x7ff8f010e190 secure :-1 s=READY pgs=1064 cs=0 l=1 rev1=1 crypto rx=0x7ff8e0009ca0 tx=0x7ff8e002c0f0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 -- 172.21.15.73:0/2072885092 shutdown_connections 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2072885092 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f01065c0 0x7ff8f010e190 unknown :-1 s=CLOSED pgs=1064 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2072885092 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f0101ea0 0x7ff8f0105ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2072885092 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01014f0 0x7ff8f01018d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 -- 172.21.15.73:0/2072885092 >> 172.21.15.73:0/2072885092 conn(0x7ff8f00786f0 msgr2=0x7ff8f0078b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:05.165 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 -- 172.21.15.73:0/2072885092 shutdown_connections 2024-03-09T00:18:05.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 -- 172.21.15.73:0/2072885092 wait complete. 2024-03-09T00:18:05.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.165+0000 7ff8f5243640 1 Processor -- start 2024-03-09T00:18:05.166 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 -- start start 2024-03-09T00:18:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f01014f0 0x7ff8f019d810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 0x7ff8f019dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.168 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f01116f0 con 0x7ff8f0197a10 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0111570 con 0x7ff8f01014f0 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8f5243640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8f0111870 con 0x7ff8f01065c0 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36844/0 (socket says 172.21.15.73:36844) 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 -- 172.21.15.73:0/2042783380 learned_addr learned my addr 172.21.15.73:0/2042783380 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8eed9d640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f01014f0 0x7ff8f019d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.169 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ee59c640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 0x7ff8f019dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 -- 172.21.15.73:0/2042783380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 msgr2=0x7ff8f019dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 0x7ff8f019dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 -- 172.21.15.73:0/2042783380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f01014f0 msgr2=0x7ff8f019d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f01014f0 0x7ff8f019d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 -- 172.21.15.73:0/2042783380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8f0198530 con 0x7ff8f0197a10 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ee59c640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 0x7ff8f019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.166+0000 7ff8ef59e640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff8e002c700 tx=0x7ff8e0032f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:05.170 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.167+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e0039040 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.167+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8f01987c0 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.167+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8e0044070 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.167+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8e00042a0 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.167+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8f01a49c0 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.168+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8f0068530 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.169+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7ff8e0004440 con 0x7ff8f0197a10 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.169+0000 7ff8d7fff640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 0x7ff8d80a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.171 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.169+0000 7ff8eed9d640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 0x7ff8d80a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.169+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(105..105 src has 1..105) v4 ==== 9613+0+0 (secure 0 0 0) 0x7ff8e00f1dd0 con 0x7ff8f0197a10 2024-03-09T00:18:05.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.170+0000 7ff8eed9d640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 0x7ff8d80a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0106170 tx=0x7ff8e800b390 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:05.172 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.173+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7ff8e00be920 con 0x7ff8f0197a10 2024-03-09T00:18:05.385 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.386+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff8f01097f0 con 0x7ff8f0197a10 2024-03-09T00:18:05.387 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-mon[32309]: osdmap e105: 12 total, 12 up, 12 in 2024-03-09T00:18:05.387 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2834224608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.387 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1698694821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.388 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/612904883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.388 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-mon[32309]: pgmap v351: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:18:05.388 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.389+0000 7ff8d7fff640 1 -- 172.21.15.73:0/2042783380 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff8e00be220 con 0x7ff8f0197a10 2024-03-09T00:18:05.388 INFO:teuthology.orchestra.run.smithi073.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-09T00:18:05.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 msgr2=0x7ff8d80a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 0x7ff8d80a4e30 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7ff8f0106170 tx=0x7ff8e800b390 comp rx=0 tx=0).stop 2024-03-09T00:18:05.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 msgr2=0x7ff8f0197ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 crypto rx=0x7ff8e002c700 tx=0x7ff8e0032f20 comp rx=0 tx=0).stop 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 shutdown_connections 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7ff8d80a2970 0x7ff8d80a4e30 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7ff8f0197a10 0x7ff8f0197ec0 unknown :-1 s=CLOSED pgs=1065 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7ff8f01065c0 0x7ff8f019dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 --2- 172.21.15.73:0/2042783380 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7ff8f01014f0 0x7ff8f019d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 >> 172.21.15.73:0/2042783380 conn(0x7ff8f00786f0 msgr2=0x7ff8f00ff3b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:05.392 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 shutdown_connections 2024-03-09T00:18:05.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.392+0000 7ff8f5243640 1 -- 172.21.15.73:0/2042783380 wait complete. 2024-03-09T00:18:05.402 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-09T00:18:05.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- 172.21.15.73:0/2861061657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 msgr2=0x7f1e8410d070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.508 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/2861061657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e8410d070 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 crypto rx=0x7f1e7c009ca0 tx=0x7f1e7c02c0f0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- 172.21.15.73:0/2861061657 shutdown_connections 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/2861061657 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e8410d070 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/2861061657 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e84100d80 0x7f1e84104dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/2861061657 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e841003d0 0x7f1e841007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- 172.21.15.73:0/2861061657 >> 172.21.15.73:0/2861061657 conn(0x7f1e840772f0 msgr2=0x7f1e84077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- 172.21.15.73:0/2861061657 shutdown_connections 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- 172.21.15.73:0/2861061657 wait complete. 2024-03-09T00:18:05.509 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 Processor -- start 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- start start 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e841003d0 0x7f1e841989a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e84199420 0x7f1e841a3a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e84110540 con 0x7f1e841054a0 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841103c0 con 0x7f1e841003d0 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.507+0000 7f1e8afb9640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e841106c0 con 0x7f1e84199420 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.510 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36866/0 (socket says 172.21.15.73:36866) 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 -- 172.21.15.73:0/1816088487 learned_addr learned my addr 172.21.15.73:0/1816088487 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e88d55640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e841003d0 0x7f1e841989a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e89556640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e84199420 0x7f1e841a3a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 -- 172.21.15.73:0/1816088487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e84199420 msgr2=0x7f1e841a3a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e84199420 0x7f1e841a3a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 -- 172.21.15.73:0/1816088487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e841003d0 msgr2=0x7f1e841989a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e841003d0 0x7f1e841989a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:05.511 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 -- 172.21.15.73:0/1816088487 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e841a41c0 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e83fff640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000e9c0 tx=0x7f1e7000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e7000cd80 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e70005500 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e70010690 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e841a44b0 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.508+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e841a4a00 con 0x7f1e841054a0 2024-03-09T00:18:05.512 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.509+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e840680e0 con 0x7f1e841054a0 2024-03-09T00:18:05.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.510+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7f1e700046c0 con 0x7f1e841054a0 2024-03-09T00:18:05.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.511+0000 7f1e81ffb640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 0x7f1e740a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:05.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.511+0000 7f1e88d55640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 0x7f1e740a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:05.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.511+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9611+0+0 (secure 0 0 0) 0x7f1e700cf560 con 0x7f1e841054a0 2024-03-09T00:18:05.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.511+0000 7f1e88d55640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 0x7f1e740a4e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e78005d00 tx=0x7f1e7800b410 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:05.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.515+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7f1e7009be80 con 0x7f1e841054a0 2024-03-09T00:18:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mon[36637]: osdmap e105: 12 total, 12 up, 12 in 2024-03-09T00:18:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2834224608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1698694821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/612904883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mon[36637]: pgmap v351: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:18:05.587 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: ignoring --setuser ceph since I am not root 2024-03-09T00:18:05.587 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: ignoring --setgroup ceph since I am not root 2024-03-09T00:18:05.587 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-mds[67595]: -- 172.21.15.179:0/3421843896 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b5e9221a40 con 0x55b5e91f4c00 2024-03-09T00:18:05.587 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:18:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: starting mds.i at 2024-03-09T00:18:05.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a[32305]: 2024-03-09T00:18:05.387+0000 7efc6db6f700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-09T00:18:05.727 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:05.728+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f1e84100d80 con 0x7f1e841054a0 2024-03-09T00:18:05.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mon[36106]: osdmap e105: 12 total, 12 up, 12 in 2024-03-09T00:18:05.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2834224608' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1698694821' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/612904883' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-09T00:18:05.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mon[36106]: pgmap v351: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 53 KiB/s wr, 22 op/s 2024-03-09T00:18:05.764 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: ignoring --setuser ceph since I am not root 2024-03-09T00:18:05.765 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: ignoring --setgroup ceph since I am not root 2024-03-09T00:18:05.765 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-mds[66501]: -- 172.21.15.102:0/1640141952 <== mon.1 v2:172.21.15.179:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55cfef919a40 con 0x55cfef8ecc00 2024-03-09T00:18:05.765 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:18:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: starting mds.b at 2024-03-09T00:18:06.389 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.390+0000 7f1e81ffb640 1 -- 172.21.15.73:0/1816088487 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1e7009b780 con 0x7f1e841054a0 2024-03-09T00:18:06.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.393+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 msgr2=0x7f1e740a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:06.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.393+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 0x7f1e740a4e60 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e78005d00 tx=0x7f1e7800b410 comp rx=0 tx=0).stop 2024-03-09T00:18:06.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.393+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 msgr2=0x7f1e84198ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:06.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.393+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 crypto rx=0x7f1e7000e9c0 tx=0x7f1e7000ee80 comp rx=0 tx=0).stop 2024-03-09T00:18:06.393 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 shutdown_connections 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7f1e84199420 0x7f1e841a3a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7f1e740a29a0 0x7f1e740a4e60 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7f1e841054a0 0x7f1e84198ee0 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 --2- 172.21.15.73:0/1816088487 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7f1e841003d0 0x7f1e841989a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 >> 172.21.15.73:0/1816088487 conn(0x7f1e840772f0 msgr2=0x7f1e840fe2b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 shutdown_connections 2024-03-09T00:18:06.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.394+0000 7f1e8afb9640 1 -- 172.21.15.73:0/1816088487 wait complete. 2024-03-09T00:18:06.405 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-09T00:18:06.513 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- 172.21.15.73:0/3173439460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa578100d80 msgr2=0x7fa578104dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- 172.21.15.73:0/3173439460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa578100d80 0x7fa578104dc0 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 crypto rx=0x7fa568009900 tx=0x7fa56802bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- 172.21.15.73:0/3173439460 shutdown_connections 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- 172.21.15.73:0/3173439460 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5781054a0 0x7fa57810d070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- 172.21.15.73:0/3173439460 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa578100d80 0x7fa578104dc0 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- 172.21.15.73:0/3173439460 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781003d0 0x7fa5781007b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- 172.21.15.73:0/3173439460 >> 172.21.15.73:0/3173439460 conn(0x7fa5780772f0 msgr2=0x7fa578077700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:06.514 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- 172.21.15.73:0/3173439460 shutdown_connections 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- 172.21.15.73:0/3173439460 wait complete. 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 Processor -- start 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 -- start start 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.512+0000 7fa5802c6640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781054a0 0x7fa57819dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5781979e0 0x7fa578197e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:06.515 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578110530 con 0x7fa5781003d0 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781103b0 con 0x7fa5781979e0 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781106b0 con 0x7fa5781054a0 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36894/0 (socket says 172.21.15.73:36894) 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 -- 172.21.15.73:0/1277515868 learned_addr learned my addr 172.21.15.73:0/1277515868 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57d861640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781054a0 0x7fa57819dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 -- 172.21.15.73:0/1277515868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781054a0 msgr2=0x7fa57819dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:06.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781054a0 0x7fa57819dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 -- 172.21.15.73:0/1277515868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5781979e0 msgr2=0x7fa578197e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5781979e0 0x7fa578197e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 -- 172.21.15.73:0/1277515868 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578198500 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa57e062640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa57400d770 tx=0x7fa57400dc30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574017070 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5781987f0 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa574005500 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa574004e00 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.513+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5781a49f0 con 0x7fa5781003d0 2024-03-09T00:18:06.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.514+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5780680e0 con 0x7fa5781003d0 2024-03-09T00:18:06.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.515+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fa574004fa0 con 0x7fa5781003d0 2024-03-09T00:18:06.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.516+0000 7fa56f7fe640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 0x7fa5640a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:06.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.516+0000 7fa57d861640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 0x7fa5640a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:06.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.516+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9611+0+0 (secure 0 0 0) 0x7fa5740cf130 con 0x7fa5781003d0 2024-03-09T00:18:06.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.516+0000 7fa57d861640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 0x7fa5640a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa568009230 tx=0x7fa568005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:06.519 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.519+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fa57409ba50 con 0x7fa5781003d0 2024-03-09T00:18:06.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-09T00:18:06.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-09T00:18:06.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-09T00:18:06.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-09T00:18:06.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: osdmap e106: 12 total, 12 up, 12 in 2024-03-09T00:18:06.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-09T00:18:06.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:06 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-09T00:18:06.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-09T00:18:06.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-09T00:18:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-09T00:18:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-09T00:18:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: osdmap e106: 12 total, 12 up, 12 in 2024-03-09T00:18:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-09T00:18:06.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:06 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-09T00:18:06.693 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:06.694+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa578100ea0 con 0x7fa5781003d0 2024-03-09T00:18:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-09T00:18:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-09T00:18:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-09T00:18:06.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2042783380' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-09T00:18:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: osdmap e106: 12 total, 12 up, 12 in 2024-03-09T00:18:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: fsmap cephfs:0/1 10 up:standby, 1 failed 2024-03-09T00:18:06.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:06 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-09T00:18:07.391 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.392+0000 7fa56f7fe640 1 -- 172.21.15.73:0/1277515868 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v107) v1 ==== 181+0+0 (secure 0 0 0) 0x7fa57409b350 con 0x7fa5781003d0 2024-03-09T00:18:07.391 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_metadata' removed 2024-03-09T00:18:07.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.395+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 msgr2=0x7fa5640a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:07.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.395+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 0x7fa5640a4e30 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fa568009230 tx=0x7fa568005fd0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.394 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.395+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 msgr2=0x7fa57819d830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:07.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.395+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 crypto rx=0x7fa57400d770 tx=0x7fa57400dc30 comp rx=0 tx=0).stop 2024-03-09T00:18:07.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 shutdown_connections 2024-03-09T00:18:07.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fa5781979e0 0x7fa578197e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.395 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fa5781054a0 0x7fa57819dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fa5640a2970 0x7fa5640a4e30 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 --2- 172.21.15.73:0/1277515868 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fa5781003d0 0x7fa57819d830 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 >> 172.21.15.73:0/1277515868 conn(0x7fa5780772f0 msgr2=0x7fa578104180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:07.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 shutdown_connections 2024-03-09T00:18:07.396 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.396+0000 7fa5802c6640 1 -- 172.21.15.73:0/1277515868 wait complete. 2024-03-09T00:18:07.407 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-09T00:18:07.516 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.516+0000 7fd5d3589640 1 -- 172.21.15.73:0/2158101051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc100d90 msgr2=0x7fd5cc104dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:07.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.516+0000 7fd5d3589640 1 --2- 172.21.15.73:0/2158101051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc100d90 0x7fd5cc104dd0 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc009900 tx=0x7fd5bc02bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:07.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- 172.21.15.73:0/2158101051 shutdown_connections 2024-03-09T00:18:07.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- 172.21.15.73:0/2158101051 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 0x7fd5cc10d080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- 172.21.15.73:0/2158101051 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc100d90 0x7fd5cc104dd0 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.517 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- 172.21.15.73:0/2158101051 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc1007c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- 172.21.15.73:0/2158101051 >> 172.21.15.73:0/2158101051 conn(0x7fd5cc0773a0 msgr2=0x7fd5cc0777b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- 172.21.15.73:0/2158101051 shutdown_connections 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- 172.21.15.73:0/2158101051 wait complete. 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 Processor -- start 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- start start 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 0x7fd5cc19dd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:07.518 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 0x7fd5cc197e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:07.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc110540 con 0x7fd5cc1003e0 2024-03-09T00:18:07.520 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc1103c0 con 0x7fd5cc197970 2024-03-09T00:18:07.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.517+0000 7fd5d3589640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5cc1106c0 con 0x7fd5cc1054b0 2024-03-09T00:18:07.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:07.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:36906/0 (socket says 172.21.15.73:36906) 2024-03-09T00:18:07.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 -- 172.21.15.73:0/3133359444 learned_addr learned my addr 172.21.15.73:0/3133359444 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:07.521 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d0b24640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 0x7fd5cc19dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1b26640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 0x7fd5cc197e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 -- 172.21.15.73:0/3133359444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 msgr2=0x7fd5cc19dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 0x7fd5cc19dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 -- 172.21.15.73:0/3133359444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 msgr2=0x7fd5cc197e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 0x7fd5cc197e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 -- 172.21.15.73:0/3133359444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5cc198490 con 0x7fd5cc1003e0 2024-03-09T00:18:07.522 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1b26640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 0x7fd5cc197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d1325640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800db80 tx=0x7fd5c800b630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c8017070 con 0x7fd5cc1003e0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5c8003b70 con 0x7fd5cc1003e0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5cc198780 con 0x7fd5cc1003e0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd5c80106c0 con 0x7fd5cc1003e0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.518+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5cc1a4a20 con 0x7fd5cc1003e0 2024-03-09T00:18:07.523 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.519+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5cc0680e0 con 0x7fd5cc1003e0 2024-03-09T00:18:07.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.520+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fd5c8004020 con 0x7fd5cc1003e0 2024-03-09T00:18:07.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.521+0000 7fd5c27fc640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 0x7fd5b80a4eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:07.524 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.521+0000 7fd5d0b24640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 0x7fd5b80a4eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:07.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.521+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 9057+0+0 (secure 0 0 0) 0x7fd5c80cedd0 con 0x7fd5cc1003e0 2024-03-09T00:18:07.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.521+0000 7fd5d0b24640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 0x7fd5b80a4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000c00 tx=0x7fd5bc005fd0 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:07.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.526+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fd5c8020070 con 0x7fd5cc1003e0 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: Cluster is now healthy 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.179:6838/1486204993,v1:172.21.15.179:6839/1486204993] up:boot 2024-03-09T00:18:07.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: mds.? [v2:172.21.15.102:6838/1091085366,v1:172.21.15.102:6839/1091085366] up:boot 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: fsmap 12 up:standby 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: pgmap v353: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 28 op/s 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.587 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.588 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:07 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: Cluster is now healthy 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.179:6838/1486204993,v1:172.21.15.179:6839/1486204993] up:boot 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: mds.? [v2:172.21.15.102:6838/1091085366,v1:172.21.15.102:6839/1091085366] up:boot 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: fsmap 12 up:standby 2024-03-09T00:18:07.657 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: pgmap v353: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 28 op/s 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.658 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:07 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.700 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:07.701+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd5cc198ea0 con 0x7fd5cc1003e0 2024-03-09T00:18:07.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-09T00:18:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-09T00:18:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: Cluster is now healthy 2024-03-09T00:18:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1816088487' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-09T00:18:07.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.179:6838/1486204993,v1:172.21.15.179:6839/1486204993] up:boot 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: mds.? [v2:172.21.15.102:6838/1091085366,v1:172.21.15.102:6839/1091085366] up:boot 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: fsmap 12 up:standby 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-09T00:18:07.765 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: pgmap v353: 129 pgs: 129 active+clean; 35 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 67 KiB/s wr, 28 op/s 2024-03-09T00:18:07.766 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.766 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.766 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:07.766 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:07 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:08.403 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.404+0000 7fd5c27fc640 1 -- 172.21.15.73:0/3133359444 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v108) v1 ==== 169+0+0 (secure 0 0 0) 0x7fd5c809b910 con 0x7fd5cc1003e0 2024-03-09T00:18:08.403 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_data' removed 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 msgr2=0x7fd5b80a4eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 0x7fd5b80a4eb0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fd5bc000c00 tx=0x7fd5bc005fd0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 msgr2=0x7fd5cc19d7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 secure :-1 s=READY pgs=1071 cs=0 l=1 rev1=1 crypto rx=0x7fd5c800db80 tx=0x7fd5c800b630 comp rx=0 tx=0).stop 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 shutdown_connections 2024-03-09T00:18:08.406 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fd5cc197970 0x7fd5cc197e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fd5cc1054b0 0x7fd5cc19dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fd5b80a29f0 0x7fd5b80a4eb0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 --2- 172.21.15.73:0/3133359444 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fd5cc1003e0 0x7fd5cc19d7c0 unknown :-1 s=CLOSED pgs=1071 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 >> 172.21.15.73:0/3133359444 conn(0x7fd5cc0773a0 msgr2=0x7fd5cc0687e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 shutdown_connections 2024-03-09T00:18:08.407 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.406+0000 7fd5d3589640 1 -- 172.21.15.73:0/3133359444 wait complete. 2024-03-09T00:18:08.420 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data_ec cephfs_data_ec --yes-i-really-really-mean-it 2024-03-09T00:18:08.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 -- 172.21.15.73:0/574593762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c100410 msgr2=0x7fc26c105f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:08.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 --2- 172.21.15.73:0/574593762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c100410 0x7fc26c105f40 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 crypto rx=0x7fc25c009900 tx=0x7fc25c02bc30 comp rx=0 tx=0).stop 2024-03-09T00:18:08.531 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 -- 172.21.15.73:0/574593762 shutdown_connections 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 --2- 172.21.15.73:0/574593762 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c106670 0x7fc26c10e1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 --2- 172.21.15.73:0/574593762 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c100410 0x7fc26c105f40 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 --2- 172.21.15.73:0/574593762 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c0ffa60 0x7fc26c0ffe40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 -- 172.21.15.73:0/574593762 >> 172.21.15.73:0/574593762 conn(0x7fc26c0767b0 msgr2=0x7fc26c0fdb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 -- 172.21.15.73:0/574593762 shutdown_connections 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.529+0000 7fc270a9a640 1 -- 172.21.15.73:0/574593762 wait complete. 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 Processor -- start 2024-03-09T00:18:08.532 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 -- start start 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 --2- >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 0x7fc26c19d820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 --2- >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 0x7fc26c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 -- --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c111690 con 0x7fc26c1979d0 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 -- --> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c111510 con 0x7fc26c0ffa60 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc270a9a640 1 -- --> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc26c111810 con 0x7fc26c106670 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:08.533 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 --2- >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.73:3300/0 says I am v2:172.21.15.73:43388/0 (socket says 172.21.15.73:43388) 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 -- 172.21.15.73:0/2338162447 learned_addr learned my addr 172.21.15.73:0/2338162447 (peer_addr_for_me v2:172.21.15.73:0/0) 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26a59c640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 0x7fc26c19d820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc269d9b640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 0x7fc26c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 -- 172.21.15.73:0/2338162447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 msgr2=0x7fc26c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 0x7fc26c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 -- 172.21.15.73:0/2338162447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 msgr2=0x7fc26c19d820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 0x7fc26c19d820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:08.534 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26ad9d640 1 -- 172.21.15.73:0/2338162447 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc26c1984f0 con 0x7fc26c1979d0 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc269d9b640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 0x7fc26c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.530+0000 7fc26a59c640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 0x7fc26c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc26ad9d640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc26400eaa0 tx=0x7fc26400ef60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc26400cdb0 con 0x7fc26c1979d0 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc264005020 con 0x7fc26c1979d0 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc26c1987e0 con 0x7fc26c1979d0 2024-03-09T00:18:08.535 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2640116b0 con 0x7fc26c1979d0 2024-03-09T00:18:08.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.531+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26c1a49e0 con 0x7fc26c1979d0 2024-03-09T00:18:08.536 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.532+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc26c102b10 con 0x7fc26c1979d0 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.533+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 4 ==== mgrmap(e 26) v1 ==== 137421+0+0 (secure 0 0 0) 0x7fc264005560 con 0x7fc26c1979d0 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.533+0000 7fc25b7fe640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 0x7fc2540a4e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.533+0000 7fc26a59c640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 0x7fc2540a4e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.533+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 8498+0+0 (secure 0 0 0) 0x7fc2640cf4f0 con 0x7fc26c1979d0 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.534+0000 7fc26a59c640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 0x7fc2540a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26c198b10 tx=0x7fc260005e70 comp rx=0 tx=0).ready entity=mgr.15564 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-09T00:18:08.537 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.537+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+184905 (secure 0 0 0) 0x7fc264016030 con 0x7fc26c1979d0 2024-03-09T00:18:08.655 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:08.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: osdmap e107: 12 total, 12 up, 12 in 2024-03-09T00:18:08.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:08.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:18:08.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:18:08.656 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:08 smithi073 ceph-mon[32309]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:08.731 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:08.732+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 --> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc26c068530 con 0x7fc26c1979d0 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: osdmap e107: 12 total, 12 up, 12 in 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:18:08.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:08 smithi102 ceph-mon[36106]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/1277515868' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: osdmap e107: 12 total, 12 up, 12 in 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-09T00:18:08.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:08 smithi179 ceph-mon[36637]: from='mgr.15564 172.21.15.73:0/1322551151' entity='mgr.x' 2024-03-09T00:18:09.411 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.412+0000 7fc25b7fe640 1 -- 172.21.15.73:0/2338162447 <== mon.0 v2:172.21.15.73:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data_ec' removed v109) v1 ==== 178+0+0 (secure 0 0 0) 0x7fc26c068530 con 0x7fc26c1979d0 2024-03-09T00:18:09.411 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'cephfs_data_ec' removed 2024-03-09T00:18:09.413 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 msgr2=0x7fc2540a4e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 0x7fc2540a4e30 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fc26c198b10 tx=0x7fc260005e70 comp rx=0 tx=0).stop 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 msgr2=0x7fc26c197e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 secure :-1 s=READY pgs=1073 cs=0 l=1 rev1=1 crypto rx=0x7fc26400eaa0 tx=0x7fc26400ef60 comp rx=0 tx=0).stop 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 shutdown_connections 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:6800/2716694463,v1:172.21.15.73:6801/2716694463] conn(0x7fc2540a2970 0x7fc2540a4e30 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] conn(0x7fc26c1979d0 0x7fc26c197e80 unknown :-1 s=CLOSED pgs=1073 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.179:3300/0,v1:172.21.15.179:6789/0] conn(0x7fc26c106670 0x7fc26c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:09.414 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 --2- 172.21.15.73:0/2338162447 >> [v2:172.21.15.102:3300/0,v1:172.21.15.102:6789/0] conn(0x7fc26c0ffa60 0x7fc26c19d820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-09T00:18:09.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 >> 172.21.15.73:0/2338162447 conn(0x7fc26c0767b0 msgr2=0x7fc26c104a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-09T00:18:09.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 shutdown_connections 2024-03-09T00:18:09.415 INFO:teuthology.orchestra.run.smithi073.stderr:2024-03-09T00:18:09.414+0000 7fc270a9a640 1 -- 172.21.15.73:0/2338162447 wait complete. 2024-03-09T00:18:09.429 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:18:09.457 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:18:09.489 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:18:09.523 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-09T00:18:09.523 DEBUG:teuthology.orchestra.run.smithi073:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-09T00:18:09.539 DEBUG:teuthology.orchestra.run.smithi102:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-09T00:18:09.555 DEBUG:teuthology.orchestra.run.smithi179:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-09T00:18:09.579 INFO:tasks.cephadm:Stopping all daemons... 2024-03-09T00:18:09.580 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-09T00:18:09.580 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a 2024-03-09T00:18:09.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:09 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:09.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:09 smithi102 ceph-mon[36106]: osdmap e108: 12 total, 12 up, 12 in 2024-03-09T00:18:09.763 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:09 smithi102 ceph-mon[36106]: from='client.? 172.21.15.73:0/2338162447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:09.764 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:09 smithi102 ceph-mon[36106]: pgmap v356: 33 pgs: 33 active+clean; 26 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:18:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:09 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:09 smithi179 ceph-mon[36637]: osdmap e108: 12 total, 12 up, 12 in 2024-03-09T00:18:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:09 smithi179 ceph-mon[36637]: from='client.? 172.21.15.73:0/2338162447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:09.836 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:09 smithi179 ceph-mon[36637]: pgmap v356: 33 pgs: 33 active+clean; 26 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:18:09.896 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/3133359444' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]': finished 2024-03-09T00:18:09.897 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-mon[32309]: osdmap e108: 12 total, 12 up, 12 in 2024-03-09T00:18:09.897 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-mon[32309]: from='client.? 172.21.15.73:0/2338162447' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data_ec", "pool2": "cephfs_data_ec", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-09T00:18:09.897 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-mon[32309]: pgmap v356: 33 pgs: 33 active+clean; 26 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-09T00:18:09.897 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 systemd[1]: Stopping Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:10.156 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a[32305]: 2024-03-09T00:18:09.897+0000 7efc7337a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:10.156 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a[32305]: 2024-03-09T00:18:09.897+0000 7efc7337a700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-09T00:18:10.156 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 podman[105907]: 2024-03-09 00:18:10.081054661 +0000 UTC m=+0.309885616 container died 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-09T00:18:10.740 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 podman[105907]: 2024-03-09 00:18:10.507209068 +0000 UTC m=+0.736040030 container cleanup 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-09T00:18:10.741 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 bash[105907]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a 2024-03-09T00:18:10.980 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service' 2024-03-09T00:18:11.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 podman[105921]: 2024-03-09 00:18:10.742180235 +0000 UTC m=+0.663236884 container remove 7374c2d990184184803b08355841bfd0654523924953f159290771102f85dbf6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-09T00:18:11.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-09T00:18:11.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 systemd[1]: Stopped Ceph mon.a for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:18:11.005 INFO:journalctl@ceph.mon.a.smithi073.stdout:Mar 09 00:18:10 smithi073 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.a.service: Consumed 21.264s CPU time. 2024-03-09T00:18:11.026 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:11.026 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-09T00:18:11.027 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-09T00:18:11.027 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.b 2024-03-09T00:18:11.351 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 systemd[1]: Stopping Ceph mon.b for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:11.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b[36101]: 2024-03-09T00:18:11.351+0000 7faa4f33d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:11.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b[36101]: 2024-03-09T00:18:11.351+0000 7faa4f33d700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-09T00:18:11.760 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 podman[72032]: 2024-03-09 00:18:11.454406348 +0000 UTC m=+0.229382522 container died feeab1859012fcf247f8987bf9984d4d816ae53631946e0ad1eeebfb1997fe6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-09T00:18:12.013 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 podman[72032]: 2024-03-09 00:18:11.861457866 +0000 UTC m=+0.636434005 container cleanup feeab1859012fcf247f8987bf9984d4d816ae53631946e0ad1eeebfb1997fe6b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True) 2024-03-09T00:18:12.013 INFO:journalctl@ceph.mon.b.smithi102.stdout:Mar 09 00:18:11 smithi102 bash[72032]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-b 2024-03-09T00:18:12.227 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.b.service' 2024-03-09T00:18:12.279 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:12.280 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-09T00:18:12.280 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-09T00:18:12.280 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.c 2024-03-09T00:18:12.586 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:12 smithi179 systemd[1]: Stopping Ceph mon.c for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:13.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-c[36630]: 2024-03-09T00:18:12.597+0000 7fe14b2ee700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:13.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-c[36630]: 2024-03-09T00:18:12.597+0000 7fe14b2ee700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-09T00:18:13.086 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:12 smithi179 podman[73215]: 2024-03-09 00:18:12.802414354 +0000 UTC m=+0.325078222 container died a1f882149fa6e317c4bd5e6759e15e537a95aef474445d6cebcf57d71e6bec38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:13.440 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:13 smithi179 podman[73215]: 2024-03-09 00:18:13.198496164 +0000 UTC m=+0.721160026 container cleanup a1f882149fa6e317c4bd5e6759e15e537a95aef474445d6cebcf57d71e6bec38 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-c, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-09T00:18:13.440 INFO:journalctl@ceph.mon.c.smithi179.stdout:Mar 09 00:18:13 smithi179 bash[73215]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mon-c 2024-03-09T00:18:13.565 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mon.c.service' 2024-03-09T00:18:13.616 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:13.617 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-09T00:18:13.617 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-09T00:18:13.617 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x 2024-03-09T00:18:13.905 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:13 smithi073 systemd[1]: Stopping Ceph mgr.x for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:14.228 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x[32539]: 2024-03-09T00:18:13.919+0000 7fbf8a04a700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:14.228 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:13 smithi073 podman[106029]: 2024-03-09 00:18:13.950940213 +0000 UTC m=+0.141736878 container died 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-09T00:18:14.588 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 podman[106029]: 2024-03-09 00:18:14.337870139 +0000 UTC m=+0.528666804 container cleanup 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-09T00:18:14.588 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 bash[106029]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x 2024-03-09T00:18:14.819 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service' 2024-03-09T00:18:14.843 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 podman[106043]: 2024-03-09 00:18:14.589952958 +0000 UTC m=+0.637051439 container remove 3d5bbc20058d43a159696deae2cf52bba6232aad52363dc4d604a402cb3adb89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:7d63482aa89628d733e298984a762389b625c761, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-x, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-09T00:18:14.843 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service: Deactivated successfully. 2024-03-09T00:18:14.843 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 systemd[1]: Stopped Ceph mgr.x for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:18:14.843 INFO:journalctl@ceph.mgr.x.smithi073.stdout:Mar 09 00:18:14 smithi073 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.x.service: Consumed 1min 23.220s CPU time. 2024-03-09T00:18:14.857 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:14.857 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-09T00:18:14.857 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-09T00:18:14.857 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.y 2024-03-09T00:18:15.194 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:18:14 smithi102 systemd[1]: Stopping Ceph mgr.y for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:15.507 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:18:15 smithi102 podman[72153]: 2024-03-09 00:18:15.195495302 +0000 UTC m=+0.151088087 container died 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-09T00:18:15.763 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:18:15 smithi102 podman[72153]: 2024-03-09 00:18:15.608696899 +0000 UTC m=+0.564289674 container cleanup 5139b0975ba276ffc56003e2fc4942fc6d797eb8a8c45af0d4e1bebd76a828a2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-09T00:18:15.763 INFO:journalctl@ceph.mgr.y.smithi102.stdout:Mar 09 00:18:15 smithi102 bash[72153]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-y 2024-03-09T00:18:15.979 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.y.service' 2024-03-09T00:18:16.017 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:16.017 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-09T00:18:16.017 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-09T00:18:16.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z 2024-03-09T00:18:16.336 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 systemd[1]: Stopping Ceph mgr.z for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:16.628 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 podman[73339]: 2024-03-09 00:18:16.341148931 +0000 UTC m=+0.134938244 container died b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:18:16.937 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 podman[73339]: 2024-03-09 00:18:16.720780307 +0000 UTC m=+0.514569630 container cleanup b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-09T00:18:16.938 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 bash[73339]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z 2024-03-09T00:18:17.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z.service' 2024-03-09T00:18:17.201 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 podman[73351]: 2024-03-09 00:18:16.939292698 +0000 UTC m=+0.595125011 container remove b45a969a22500b172a5fcaf71bbab695c5e11d66599811914794a5faf9f8b319 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mgr-z, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2) 2024-03-09T00:18:17.201 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:16 smithi179 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z.service: Main process exited, code=exited, status=143/n/a 2024-03-09T00:18:17.201 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:17 smithi179 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z.service: Failed with result 'exit-code'. 2024-03-09T00:18:17.201 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:17 smithi179 systemd[1]: Stopped Ceph mgr.z for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:18:17.201 INFO:journalctl@ceph.mgr.z.smithi179.stdout:Mar 09 00:18:17 smithi179 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mgr.z.service: Consumed 20.603s CPU time. 2024-03-09T00:18:17.214 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:17.214 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-09T00:18:17.214 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-09T00:18:17.215 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.0 2024-03-09T00:18:17.533 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:17 smithi073 systemd[1]: Stopping Ceph osd.0 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:17.905 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0[46677]: 2024-03-09T00:18:17.534+0000 7f96f1a27700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:17.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0[46677]: 2024-03-09T00:18:17.534+0000 7f96f1a27700 -1 osd.0 109 *** Got signal Terminated *** 2024-03-09T00:18:17.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0[46677]: 2024-03-09T00:18:17.534+0000 7f96f1a27700 -1 osd.0 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:23.276 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:22 smithi073 podman[106153]: 2024-03-09 00:18:22.984508649 +0000 UTC m=+5.583051783 container died 75e4558d67570cb3275a67e849246a2591126e3f7be8ac3b6058b8cdac61de86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307) 2024-03-09T00:18:23.593 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:23 smithi073 podman[106153]: 2024-03-09 00:18:23.376355817 +0000 UTC m=+5.974898948 container cleanup 75e4558d67570cb3275a67e849246a2591126e3f7be8ac3b6058b8cdac61de86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-09T00:18:23.593 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:23 smithi073 bash[106153]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0 2024-03-09T00:18:23.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:23 smithi073 podman[106170]: 2024-03-09 00:18:23.594516064 +0000 UTC m=+0.613880255 container remove 75e4558d67570cb3275a67e849246a2591126e3f7be8ac3b6058b8cdac61de86 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS) 2024-03-09T00:18:24.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:23 smithi073 podman[106258]: 2024-03-09 00:18:23.895325303 +0000 UTC m=+0.018291248 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:18:24.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:24 smithi073 podman[106258]: 2024-03-09 00:18:24.00404615 +0000 UTC m=+0.127012123 container create 52ddaca41ecf2e81cbcb4d45ac49abd4ef555e3ede71ff1ac232f6020943450c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0-deactivate, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:24.264 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:24 smithi073 podman[106258]: 2024-03-09 00:18:24.265306339 +0000 UTC m=+0.388272266 container init 52ddaca41ecf2e81cbcb4d45ac49abd4ef555e3ede71ff1ac232f6020943450c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0-deactivate, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-09T00:18:24.556 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:24 smithi073 podman[106258]: 2024-03-09 00:18:24.268619656 +0000 UTC m=+0.391585594 container start 52ddaca41ecf2e81cbcb4d45ac49abd4ef555e3ede71ff1ac232f6020943450c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0-deactivate, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:18:24.556 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:24 smithi073 podman[106258]: 2024-03-09 00:18:24.332014368 +0000 UTC m=+0.454980316 container attach 52ddaca41ecf2e81cbcb4d45ac49abd4ef555e3ede71ff1ac232f6020943450c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-09T00:18:24.906 INFO:journalctl@ceph.osd.0.smithi073.stdout:Mar 09 00:18:24 smithi073 podman[106258]: 2024-03-09 00:18:24.557863816 +0000 UTC m=+0.680829735 container died 52ddaca41ecf2e81cbcb4d45ac49abd4ef555e3ede71ff1ac232f6020943450c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:18:25.296 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.0.service' 2024-03-09T00:18:25.344 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:25.344 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-09T00:18:25.344 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-09T00:18:25.344 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.1 2024-03-09T00:18:25.637 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:25 smithi102 systemd[1]: Stopping Ceph osd.1 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:26.013 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1[40730]: 2024-03-09T00:18:25.637+0000 7ff01ffc3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:26.013 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1[40730]: 2024-03-09T00:18:25.637+0000 7ff01ffc3700 -1 osd.1 109 *** Got signal Terminated *** 2024-03-09T00:18:26.014 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1[40730]: 2024-03-09T00:18:25.637+0000 7ff01ffc3700 -1 osd.1 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:31.263 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:31 smithi102 podman[72289]: 2024-03-09 00:18:31.009033786 +0000 UTC m=+5.481207037 container died 0017561d8b1cb90cae6bea34908d7752ceedf6ad0f65c493c08553f53e2c9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:31.763 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:31 smithi102 podman[72289]: 2024-03-09 00:18:31.44714818 +0000 UTC m=+5.919321417 container cleanup 0017561d8b1cb90cae6bea34908d7752ceedf6ad0f65c493c08553f53e2c9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS) 2024-03-09T00:18:31.764 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:31 smithi102 bash[72289]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1 2024-03-09T00:18:32.263 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:31 smithi102 podman[72307]: 2024-03-09 00:18:31.923828749 +0000 UTC m=+0.911522811 container remove 0017561d8b1cb90cae6bea34908d7752ceedf6ad0f65c493c08553f53e2c9c1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:32.684 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.315444867 +0000 UTC m=+0.018093862 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:18:32.684 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.441700554 +0000 UTC m=+0.144349541 container create ca1145a1223e1b1b31f4e4fb648ea4063ddc8abe1d0da0428d1d2a25bb8f3635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-09T00:18:32.685 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.685967079 +0000 UTC m=+0.388616086 container init ca1145a1223e1b1b31f4e4fb648ea4063ddc8abe1d0da0428d1d2a25bb8f3635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:32.977 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.689103054 +0000 UTC m=+0.391752056 container start ca1145a1223e1b1b31f4e4fb648ea4063ddc8abe1d0da0428d1d2a25bb8f3635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_BRANCH=HEAD) 2024-03-09T00:18:32.977 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.786230074 +0000 UTC m=+0.488879073 container attach ca1145a1223e1b1b31f4e4fb648ea4063ddc8abe1d0da0428d1d2a25bb8f3635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:18:33.263 INFO:journalctl@ceph.osd.1.smithi102.stdout:Mar 09 00:18:32 smithi102 podman[72395]: 2024-03-09 00:18:32.978135896 +0000 UTC m=+0.680784901 container died ca1145a1223e1b1b31f4e4fb648ea4063ddc8abe1d0da0428d1d2a25bb8f3635 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:18:33.599 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.1.service' 2024-03-09T00:18:33.640 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:33.640 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-09T00:18:33.641 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-09T00:18:33.641 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.2 2024-03-09T00:18:33.949 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:33 smithi179 systemd[1]: Stopping Ceph osd.2 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:34.336 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2[41807]: 2024-03-09T00:18:33.949+0000 7f8da5e7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:34.336 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2[41807]: 2024-03-09T00:18:33.949+0000 7f8da5e7b700 -1 osd.2 109 *** Got signal Terminated *** 2024-03-09T00:18:34.336 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2[41807]: 2024-03-09T00:18:33.949+0000 7f8da5e7b700 -1 osd.2 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:39.586 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:39 smithi179 podman[73474]: 2024-03-09 00:18:39.270987907 +0000 UTC m=+5.434684025 container died e04dd2f6fb3b049d44fa83d2971314921ac855893915500a242248f2ffffce4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-09T00:18:39.928 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:39 smithi179 podman[73474]: 2024-03-09 00:18:39.659732248 +0000 UTC m=+5.823428359 container cleanup e04dd2f6fb3b049d44fa83d2971314921ac855893915500a242248f2ffffce4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-09T00:18:39.928 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:39 smithi179 bash[73474]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2 2024-03-09T00:18:40.263 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:39 smithi179 podman[73491]: 2024-03-09 00:18:39.929379778 +0000 UTC m=+0.656657966 container remove e04dd2f6fb3b049d44fa83d2971314921ac855893915500a242248f2ffffce4d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:18:40.574 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.168409463 +0000 UTC m=+0.018350544 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:18:40.574 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.289317156 +0000 UTC m=+0.139258275 container create 57e34fceb94a5bdea559ab8d072490f4624c58aa43d09a89ad1dd45c97b1da1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS) 2024-03-09T00:18:40.574 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.483539202 +0000 UTC m=+0.333480288 container init 57e34fceb94a5bdea559ab8d072490f4624c58aa43d09a89ad1dd45c97b1da1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:18:40.574 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.486813433 +0000 UTC m=+0.336754513 container start 57e34fceb94a5bdea559ab8d072490f4624c58aa43d09a89ad1dd45c97b1da1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:18:40.575 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.575328324 +0000 UTC m=+0.425269409 container attach 57e34fceb94a5bdea559ab8d072490f4624c58aa43d09a89ad1dd45c97b1da1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:40.836 INFO:journalctl@ceph.osd.2.smithi179.stdout:Mar 09 00:18:40 smithi179 podman[73579]: 2024-03-09 00:18:40.776410458 +0000 UTC m=+0.626351529 container died 57e34fceb94a5bdea559ab8d072490f4624c58aa43d09a89ad1dd45c97b1da1a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:18:41.459 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.2.service' 2024-03-09T00:18:41.502 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:41.502 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-09T00:18:41.503 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-09T00:18:41.503 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.3 2024-03-09T00:18:41.829 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:41 smithi073 systemd[1]: Stopping Ceph osd.3 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:42.089 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:41.829+0000 7f0b5e8da700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:42.090 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:41.829+0000 7f0b5e8da700 -1 osd.3 109 *** Got signal Terminated *** 2024-03-09T00:18:42.090 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:41 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:41.829+0000 7f0b5e8da700 -1 osd.3 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:44.406 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:43 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:43.952+0000 7f0b5aece700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.780947+0000 front 2024-03-09T00:18:21.780906+0000 (oldest deadline 2024-03-09T00:18:43.481128+0000) 2024-03-09T00:18:44.586 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:44.252+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:45.405 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:44 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:44.989+0000 7f0b5aece700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.780947+0000 front 2024-03-09T00:18:21.780906+0000 (oldest deadline 2024-03-09T00:18:43.481128+0000) 2024-03-09T00:18:45.586 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:45.203+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:46.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:46.241+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:46.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:46.016+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:46.406 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:45 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3[53638]: 2024-03-09T00:18:45.950+0000 7f0b5aece700 -1 osd.3 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.780947+0000 front 2024-03-09T00:18:21.780906+0000 (oldest deadline 2024-03-09T00:18:43.481128+0000) 2024-03-09T00:18:46.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:46 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:46.220+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:47.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:47.195+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:47.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:46.985+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:47.488 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:47 smithi073 podman[106368]: 2024-03-09 00:18:47.205612193 +0000 UTC m=+5.508020349 container died 0eced198d432fcbb5803b2e5131f8f9fbfd2f8256eebbe06d35c3346a33bad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:18:47.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:47 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:47.257+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:47.871 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:47 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:47.510+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:47.872 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:47 smithi073 podman[106368]: 2024-03-09 00:18:47.579948312 +0000 UTC m=+5.882356469 container cleanup 0eced198d432fcbb5803b2e5131f8f9fbfd2f8256eebbe06d35c3346a33bad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307) 2024-03-09T00:18:47.872 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:47 smithi073 bash[106368]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3 2024-03-09T00:18:48.155 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:47 smithi073 podman[106399]: 2024-03-09 00:18:47.873138458 +0000 UTC m=+0.661192370 container remove 0eced198d432fcbb5803b2e5131f8f9fbfd2f8256eebbe06d35c3346a33bad33 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux ) 2024-03-09T00:18:48.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:48.024+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:48.338 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:48.215+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:48.486 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.109382628 +0000 UTC m=+0.017881532 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:18:48.486 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.257503418 +0000 UTC m=+0.166002303 container create d0177f8a5a6ba714b5a5b803b161cf7f6c9abf840cfe7aa2d67e037b542303bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3-deactivate, ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:18:48.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:48 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:48.226+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:48.823 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.535657984 +0000 UTC m=+0.444156879 container init d0177f8a5a6ba714b5a5b803b161cf7f6c9abf840cfe7aa2d67e037b542303bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:48.823 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.539282803 +0000 UTC m=+0.447781689 container start d0177f8a5a6ba714b5a5b803b161cf7f6c9abf840cfe7aa2d67e037b542303bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3-deactivate, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.build-date=20240307, ceph=True, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-09T00:18:48.823 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.643825011 +0000 UTC m=+0.552323935 container attach d0177f8a5a6ba714b5a5b803b161cf7f6c9abf840cfe7aa2d67e037b542303bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-09T00:18:48.824 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:48 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:48.487+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:49.155 INFO:journalctl@ceph.osd.3.smithi073.stdout:Mar 09 00:18:48 smithi073 podman[106491]: 2024-03-09 00:18:48.824651286 +0000 UTC m=+0.733150181 container died d0177f8a5a6ba714b5a5b803b161cf7f6c9abf840cfe7aa2d67e037b542303bb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-3-deactivate, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-09T00:18:49.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:49.241+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:49.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:49.055+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:49.383 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.3.service' 2024-03-09T00:18:49.424 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:49.425 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-09T00:18:49.425 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-09T00:18:49.425 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.4 2024-03-09T00:18:49.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:49 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:49.531+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:49.743 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:49 smithi102 systemd[1]: Stopping Ceph osd.4 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:49.744 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:49.270+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:50.013 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4[47558]: 2024-03-09T00:18:49.743+0000 7f167d3ab700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:50.014 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4[47558]: 2024-03-09T00:18:49.743+0000 7f167d3ab700 -1 osd.4 109 *** Got signal Terminated *** 2024-03-09T00:18:50.014 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:49 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4[47558]: 2024-03-09T00:18:49.743+0000 7f167d3ab700 -1 osd.4 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:50.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:50 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:50.244+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:50.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:50 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:50.071+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:50.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:50 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:50.242+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:50.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:50 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:50.512+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:51.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:51 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:51.222+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:51.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:51 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:51.047+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:51.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:51 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:51.245+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:51.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:51 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:51.245+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:51.898 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:51.497+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:52.156 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:51 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:51.899+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:52.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:52 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:52.015+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:52.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:52 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:52.220+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:52.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:52 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:52.220+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:52.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:52 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:52.227+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:52.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:52 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:52.227+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:52.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:52 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:52.452+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:52.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:52 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:52.452+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:53.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:53 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:53.178+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:53.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:53 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:53.178+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:53.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:53 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:53.032+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:53.405 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:52 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:52.916+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:53.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:53 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:53.262+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:53.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:53 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:53.262+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:53.867 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:53 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:53.420+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:53.867 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:53 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:53.420+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:54.155 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:53 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:53.868+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:54.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:54 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:54.223+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:54.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:54 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:54.223+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:54.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:54 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:54.011+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:54.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:54 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:54.300+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:54.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:54 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:54.300+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:54.876 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:54 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:54.456+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:54.876 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:54 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:54.456+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:55.157 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:54 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:54.877+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:55.704 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:55 smithi102 podman[72501]: 2024-03-09 00:18:55.12057864 +0000 UTC m=+5.501758217 container died c255b28bec77b74ed4a622afd14433639cd5a96acad37469e785ac638958ed63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-09T00:18:55.705 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:54 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:54.979+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:55.706 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:18:55 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:18:55.176+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:18:55.706 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:55 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:55.196+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:55.706 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:55 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:55.196+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:55.707 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:55 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:55.273+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:55.707 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:55 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:55.273+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:55.708 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:18:55 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:18:55.192+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:18:55.763 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:55 smithi102 podman[72501]: 2024-03-09 00:18:55.519020717 +0000 UTC m=+5.900200284 container cleanup c255b28bec77b74ed4a622afd14433639cd5a96acad37469e785ac638958ed63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-09T00:18:55.764 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:55 smithi102 bash[72501]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4 2024-03-09T00:18:55.864 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:55 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:55.434+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:55.864 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:55 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:55.434+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:56.125 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:55 smithi102 podman[72518]: 2024-03-09 00:18:55.795432713 +0000 UTC m=+0.672490978 container remove c255b28bec77b74ed4a622afd14433639cd5a96acad37469e785ac638958ed63 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-09T00:18:56.156 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:55 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:55.864+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:56.192 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:18:56 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:18:56.169+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:18:56.193 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:55 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:55.932+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:56.193 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:55 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:55.932+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:18:56.415 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:18:56 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:18:56.197+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:18:56.415 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.029225683 +0000 UTC m=+0.017552585 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:18:56.416 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.146162029 +0000 UTC m=+0.134488914 container create 78e5566c22dcb2277bc65deb166f17b3b7dad862232777aea80a8e8429c259f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:56.416 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:56 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:56.285+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:56.416 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:56 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:56.285+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:56.586 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:56 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:56.192+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:56.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:56 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:56.192+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:56.726 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.416123291 +0000 UTC m=+0.404450196 container init 78e5566c22dcb2277bc65deb166f17b3b7dad862232777aea80a8e8429c259f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-09T00:18:56.726 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.419647002 +0000 UTC m=+0.407973893 container start 78e5566c22dcb2277bc65deb166f17b3b7dad862232777aea80a8e8429c259f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-09T00:18:56.726 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.499191256 +0000 UTC m=+0.487518156 container attach 78e5566c22dcb2277bc65deb166f17b3b7dad862232777aea80a8e8429c259f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:18:56.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:56 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:56.476+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:56.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:56 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:56.476+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:57.013 INFO:journalctl@ceph.osd.4.smithi102.stdout:Mar 09 00:18:56 smithi102 podman[72611]: 2024-03-09 00:18:56.72684341 +0000 UTC m=+0.715170304 container died 78e5566c22dcb2277bc65deb166f17b3b7dad862232777aea80a8e8429c259f8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-4-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2) 2024-03-09T00:18:57.202 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:56 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:56.939+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:57.202 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:56 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:56.939+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:18:57.402 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:18:57 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:18:57.209+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:18:57.402 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:57 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:57.286+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:57.402 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:57 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:57.286+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:57.405 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:56 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:56.914+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:57.421 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.4.service' 2024-03-09T00:18:57.466 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:18:57.467 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-09T00:18:57.467 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-09T00:18:57.467 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.5 2024-03-09T00:18:57.497 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:18:57.202+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:18:57.498 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:57.240+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:57.498 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:57.240+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:57.773 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 systemd[1]: Stopping Ceph osd.5 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:18:57.905 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:57.523+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:57.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:57.523+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:58.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:57.921+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:58.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:57.921+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:18:58.087 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:57.773+0000 7f84573a7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:18:58.087 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:57.773+0000 7f84573a7700 -1 osd.5 109 *** Got signal Terminated *** 2024-03-09T00:18:58.087 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:57 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:57.773+0000 7f84573a7700 -1 osd.5 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:18:58.156 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:57 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:57.905+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:58.513 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:18:58 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:18:58.250+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:18:58.514 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:58 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:58.313+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:58.514 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:58 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:58.313+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:58.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:18:58 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:18:58.245+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:18:58.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:58 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:58.281+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:58.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:58 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:58.281+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:58.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:58.479+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:58.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:58.479+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:18:59.286 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:58 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:58.906+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:18:59.286 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:58 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:58.906+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:18:59.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:58.932+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:18:59.513 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:18:59 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:18:59.260+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:18:59.514 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:59 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:59.348+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:18:59.514 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:18:59 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:18:59.348+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:18:59.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:18:59 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:18:59.285+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:18:59.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:59 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:59.324+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:18:59.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:18:59 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:18:59.324+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:18:59.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:59.432+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:18:59.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:18:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:18:59.432+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:00.301 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:59 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:59.917+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:00.302 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:18:59 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:18:59.917+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:00.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:18:59 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:18:59.923+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:00.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:00.334+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:00.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:00.302+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:19:00.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:00.302+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:19:00.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:00 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:00.291+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:00.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:00 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:00.380+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:00.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:00 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:00.380+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:00.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:00.454+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:00.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:00.454+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:00.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:00.454+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:01.274 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:00.878+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:01.275 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:00.878+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:01.275 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:00 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:00.878+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:01.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:00 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:00.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:01.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:01.381+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:01.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:01.381+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:01.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:01.274+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:19:01.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:01.274+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:19:01.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:01 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:01.312+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:01.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:01 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:01.406+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:01.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:01 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:01.406+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:01.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:01 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:01.406+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:01.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:01.500+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:01.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:01.500+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:01.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:01.500+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:02.264 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:01.923+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:02.264 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:01.923+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:02.265 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:01 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:01.923+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:02.405 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:01 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:01.943+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:02.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:02.355+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:02.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:02.355+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:02.587 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:02.264+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:19.391274+0000 front 2024-03-09T00:18:19.391243+0000 (oldest deadline 2024-03-09T00:18:43.391612+0000) 2024-03-09T00:19:02.588 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5[48354]: 2024-03-09T00:19:02.264+0000 7f844f993700 -1 osd.5 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:25.692499+0000 front 2024-03-09T00:18:25.692517+0000 (oldest deadline 2024-03-09T00:18:51.592565+0000) 2024-03-09T00:19:02.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:02 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:02.338+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:02.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:02 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:02.338+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:02.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:02 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:02.370+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:02.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:02 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:02.370+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:02.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:02 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:02.370+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:02.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:02.534+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:02.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:02.534+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:02.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:02.534+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:03.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:03 smithi179 podman[73708]: 2024-03-09 00:19:03.112827253 +0000 UTC m=+5.449730852 container died 72c095c36fff16b660a645fcd9cbc63c5e320caf8d6b036a5a0cc92e9fa40d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, ceph=True, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-09T00:19:03.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:02.892+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:03.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:02.892+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:03.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:02 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:02.892+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:03.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:02.986+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:03.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:02 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:02.986+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:03.743 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:03 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:03.381+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:03.744 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:03 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:03.382+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:03.745 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:03 smithi179 podman[73708]: 2024-03-09 00:19:03.466565663 +0000 UTC m=+5.803469244 container cleanup 72c095c36fff16b660a645fcd9cbc63c5e320caf8d6b036a5a0cc92e9fa40d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-09T00:19:03.745 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:03 smithi179 bash[73708]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5 2024-03-09T00:19:03.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:03 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:03.331+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:03.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:03 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:03.331+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:03.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:03 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:03.355+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:03.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:03 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:03.355+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:03.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:03 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:03.355+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:03.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:03.567+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:03.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:03.567+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:03.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:03.567+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:04.079 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:03 smithi179 podman[73724]: 2024-03-09 00:19:03.743757147 +0000 UTC m=+0.627037570 container remove 72c095c36fff16b660a645fcd9cbc63c5e320caf8d6b036a5a0cc92e9fa40d36 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux ) 2024-03-09T00:19:04.079 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:03 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:03.935+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:04.079 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:03 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:03.935+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:04.080 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:03 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:03.935+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:04.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:03.98292694 +0000 UTC m=+0.017365134 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:04.336 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:04.119728946 +0000 UTC m=+0.154167148 container create 2c3e4b5fc633d3515dec88c6f70e0835841defc17ca528b934def02ba31f92b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-09T00:19:04.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:03.964+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:04.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:03 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:03.964+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:04.667 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:04 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:04.423+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:04.667 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:04 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:04.423+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:04.667 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:04.372723328 +0000 UTC m=+0.407161513 container init 2c3e4b5fc633d3515dec88c6f70e0835841defc17ca528b934def02ba31f92b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-09T00:19:04.667 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:04.376374322 +0000 UTC m=+0.410812506 container start 2c3e4b5fc633d3515dec88c6f70e0835841defc17ca528b934def02ba31f92b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-09T00:19:04.667 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:04.455633373 +0000 UTC m=+0.490071586 container attach 2c3e4b5fc633d3515dec88c6f70e0835841defc17ca528b934def02ba31f92b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5-deactivate, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-09T00:19:04.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:04 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:04.359+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:04.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:04 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:04.359+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:04.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:04 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:04.369+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:04.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:04 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:04.369+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:04.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:04 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:04.369+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:04.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:04.599+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:04.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:04.599+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:04.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:04.599+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:04.932 INFO:journalctl@ceph.osd.5.smithi179.stdout:Mar 09 00:19:04 smithi179 podman[73811]: 2024-03-09 00:19:04.668135117 +0000 UTC m=+0.702573308 container died 2c3e4b5fc633d3515dec88c6f70e0835841defc17ca528b934def02ba31f92b2 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-5-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True) 2024-03-09T00:19:05.233 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:04 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:04.950+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:05.233 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:04 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:04.950+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:05.233 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:04 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:04.950+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:05.260 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.5.service' 2024-03-09T00:19:05.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:05.343 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-09T00:19:05.343 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-09T00:19:05.344 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.6 2024-03-09T00:19:05.375 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:04.955+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:05.375 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:04.955+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:05.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:05.448+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:05.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:05.448+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:05.656 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 systemd[1]: Stopping Ceph osd.6 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:05.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:05.586+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:05.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:05.586+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:05.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:05.586+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:05.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:05.381+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:05.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:05.381+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:05.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:05.367+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:05.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:05.367+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:05.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:05 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:05.367+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:05.951 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:05.657+0000 7fd223437700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:05.951 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:05.657+0000 7fd223437700 -1 osd.6 109 *** Got signal Terminated *** 2024-03-09T00:19:05.951 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:05.657+0000 7fd223437700 -1 osd.6 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:05.951 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:05.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:05.951 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:05 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:05.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:06.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:05.991+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:06.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:05.991+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:06.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:05 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:05.991+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:06.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:06 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:06.357+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:06.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:06 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:06.357+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:06.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:06 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:06.394+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:06.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:06 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:06.394+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:06.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:06 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:06.394+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:06.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:06 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:06.431+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:06.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:06 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:06.431+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:06.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:06 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:06.553+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:06.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:06 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:06.553+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:06.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:06 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:06.553+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:07.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:07.033+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:07.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:07.033+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:07.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:07.033+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:07.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:06 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:06.982+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:07.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:06 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:06.982+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:07.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:07 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:07.373+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:07.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:07 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:07.373+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:07.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:07 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:07.429+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:07.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:07 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:07.429+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:07.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:07 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:07.429+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:07.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:07.449+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:07.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:07.449+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:07.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:07 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:07.449+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:07.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:07 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:07.512+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:07.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:07 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:07.512+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:07.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:07 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:07.512+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:08.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:08.044+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:08.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:08.044+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:08.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:08.044+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:08.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:07 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:07.991+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:08.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:07 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:07.991+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:08.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:08.355+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:08.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:08.355+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:08.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:08.428+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:08.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:08.428+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:08.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:08.428+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:08.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:08.402+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:08.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:08.402+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:08.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:08 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:08.402+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:08.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:08 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:08.503+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:08.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:08 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:08.503+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:08.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:08 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:08.503+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:09.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:09.061+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:09.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:09.061+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:09.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:09.061+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:09.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:08 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:08.942+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:09.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:08 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:08.942+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:09.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:09.365+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:09.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:09.365+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:09.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:09.365+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:09.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:09.389+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:09.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:09.389+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:09.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:09 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:09.389+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:09.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:09.369+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:09.837 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:09.369+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:09.837 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:09 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:09.369+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:09.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:09.522+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:10.188 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:09.522+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:10.189 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:09.522+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:10.325 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:10.055+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:10.326 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:10.055+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:10.326 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:10.055+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:10.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:09.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.557092+0000 front 2024-03-09T00:18:30.557107+0000 (oldest deadline 2024-03-09T00:18:51.657136+0000) 2024-03-09T00:19:10.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:09.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.259115+0000 front 2024-03-09T00:18:36.259293+0000 (oldest deadline 2024-03-09T00:19:02.159022+0000) 2024-03-09T00:19:10.406 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:09 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6[60663]: 2024-03-09T00:19:09.951+0000 7fd21ba23700 -1 osd.6 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.760103+0000 front 2024-03-09T00:18:46.760165+0000 (oldest deadline 2024-03-09T00:19:09.660199+0000) 2024-03-09T00:19:10.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:10.325+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:10.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:10.325+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:10.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:10 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:10.325+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:10.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:10.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:10.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:10.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:10.375+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:10.375+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:10.375+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:10.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:10.375+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:10.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:10.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:10.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:10.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:10.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:10.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:10.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:10 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:10.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:11.316 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:11 smithi073 podman[106598]: 2024-03-09 00:19:11.016396361 +0000 UTC m=+5.477583713 container died 51a06e528129a3a88144627f9e61703632530517adea44e2cf8f356f017f3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-09T00:19:11.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:11.280+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:11.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:11.280+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:11.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:11.280+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:11.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:11.059+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:11.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:11.059+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:11.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:11 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:11.059+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:11.575 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:11 smithi073 podman[106598]: 2024-03-09 00:19:11.41661889 +0000 UTC m=+5.877806252 container cleanup 51a06e528129a3a88144627f9e61703632530517adea44e2cf8f356f017f3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-03-09T00:19:11.576 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:11 smithi073 bash[106598]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6 2024-03-09T00:19:11.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:11.332+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:11.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:11.332+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:11.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:11.332+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:11.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:11.374+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:11.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:11.374+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:11.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:11.374+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:11.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:11 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:11.374+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:11.906 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:11 smithi073 podman[106614]: 2024-03-09 00:19:11.676529885 +0000 UTC m=+0.657510538 container remove 51a06e528129a3a88144627f9e61703632530517adea44e2cf8f356f017f3811 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307) 2024-03-09T00:19:11.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:11 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:11.575+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:11.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:11 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:11.575+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:11.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:11 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:11.575+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:11.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:11 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:11.575+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:12.311 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:11.931480683 +0000 UTC m=+0.017858233 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:12.312 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:12.035614993 +0000 UTC m=+0.121992567 container create f838d856ed8926d5e81708b2aaa487001b6ee8577706e87a0f52f2fcd78df217 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-09T00:19:12.312 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:12.229534781 +0000 UTC m=+0.315912313 container init f838d856ed8926d5e81708b2aaa487001b6ee8577706e87a0f52f2fcd78df217 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:19:12.312 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:12.23269875 +0000 UTC m=+0.319076286 container start f838d856ed8926d5e81708b2aaa487001b6ee8577706e87a0f52f2fcd78df217 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6-deactivate, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-09T00:19:12.312 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:12.312801289 +0000 UTC m=+0.399178825 container attach f838d856ed8926d5e81708b2aaa487001b6ee8577706e87a0f52f2fcd78df217 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-09T00:19:12.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:12.235+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:12.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:12.235+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:12.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:12.235+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:12.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.027+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:12.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.027+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:12.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:12 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.027+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:12.656 INFO:journalctl@ceph.osd.6.smithi073.stdout:Mar 09 00:19:12 smithi073 podman[106705]: 2024-03-09 00:19:12.518579816 +0000 UTC m=+0.604957348 container died f838d856ed8926d5e81708b2aaa487001b6ee8577706e87a0f52f2fcd78df217 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-6-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-09T00:19:12.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:12 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:12.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:12.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:12 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:12.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:12.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:12 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:12.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:12.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:12 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:12.557+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:12.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:12.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:12.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:12.379+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:12.354+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:12.354+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:12.354+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:12.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:12 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:12.354+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:13.210 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.6.service' 2024-03-09T00:19:13.293 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:13.293 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-09T00:19:13.294 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-09T00:19:13.294 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.7 2024-03-09T00:19:13.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.999+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:13.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.999+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:13.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:12.999+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:13.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:13.206+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:13.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:13.206+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:13.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:13 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:13.206+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:13.613 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:13.346+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:13.614 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:13.346+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:13.614 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:13.346+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:13.614 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:13.346+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:13.615 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 systemd[1]: Stopping Ceph osd.7 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:13.615 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.367+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:13.615 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.367+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:13.615 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.367+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:13.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:13.571+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:13.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:13.571+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:13.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:13.571+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:13.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:13 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:13.571+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:14.013 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.613+0000 7f5e8fb66700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:14.014 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.613+0000 7f5e8fb66700 -1 osd.7 109 *** Got signal Terminated *** 2024-03-09T00:19:14.014 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:13.613+0000 7f5e8fb66700 -1 osd.7 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:14.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:14.008+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:14.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:14.008+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:14.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:14.008+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:14.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:14.253+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:14.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:14.253+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:14.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:14 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:14.253+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:14.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:14 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:14.597+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:14.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:14 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:14.597+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:14.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:14 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:14.597+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:14.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:14 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:14.597+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:14.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:14.382+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:14.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:14.382+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:14.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:14.382+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:14.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:14.382+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:14.766 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:14.333+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:14.766 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:14.333+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:14.766 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:14 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:14.333+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:15.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:15.239+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:15.336 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:15.239+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:15.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:15.239+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:15.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:15.004+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:15.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:15.004+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:15.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:15 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:15.004+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:15.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:15.349+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:15.765 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:15.349+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:15.765 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:15.349+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:15.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:15.415+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:15.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:15.415+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:15.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:15.415+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:15.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:15 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:15.415+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:15.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:15 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:15.606+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:15.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:15 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:15.606+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:15.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:15 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:15.606+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:15.907 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:15 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:15.606+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:16.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:16.050+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:16.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:16.050+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:16.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:16.050+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:16.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:16.221+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:16.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:16.221+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:16.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:16.221+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:16.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:16 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:16.221+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:16.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:16.355+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:16.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:16.355+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:16.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:16.355+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:16.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:16.438+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:16.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:16.438+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:16.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:16.438+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:16.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:16.438+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:16.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:16 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:16.438+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:16.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:16 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:16.639+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:16.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:16 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:16.639+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:16.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:16 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:16.639+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:16.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:16 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:16.639+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:17.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.023+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:17.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.023+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:17.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.023+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:17.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:17.198+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:17.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:17.198+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:17.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:17.198+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:17.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:17.198+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:17.763 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:17.310+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:17.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:17.310+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:17.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:17.310+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:17.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:17.416+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:17.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:17.416+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:17.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:17.416+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:17.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:17.416+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:17.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:17 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:17.416+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:17.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:17.636+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:17.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:17.636+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:17.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:17.636+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:17.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:17.636+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:17.906 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:17 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:17.636+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:18.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.985+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:18.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.985+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:18.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:17 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:17.985+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:18.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:18.155+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:18.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:18.155+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:18.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:18.155+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:18.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:18.155+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:18.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:18.294+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.270343+0000 front 2024-03-09T00:18:30.270299+0000 (oldest deadline 2024-03-09T00:18:54.271072+0000) 2024-03-09T00:19:18.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:18.294+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:38.173427+0000 front 2024-03-09T00:18:38.173282+0000 (oldest deadline 2024-03-09T00:19:01.673234+0000) 2024-03-09T00:19:18.764 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7[54158]: 2024-03-09T00:19:18.294+0000 7f5e88152700 -1 osd.7 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:46.274920+0000 front 2024-03-09T00:18:46.274765+0000 (oldest deadline 2024-03-09T00:19:09.174863+0000) 2024-03-09T00:19:18.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:18.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:18.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:18.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:18.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:18.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:18.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:18.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:18.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:18.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:19.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:18.676+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:19.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:18.676+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:19.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:18.676+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:19.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:18.676+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:19.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:18 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:18.676+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:19.322 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:19 smithi102 podman[72718]: 2024-03-09 00:19:19.048971478 +0000 UTC m=+5.561926655 container died e0cb21bb69f5b1ec602d280106c3c10c33df7fa3dcef72da146707411e915f71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:19:19.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:18.958+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:19.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:18.958+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:19.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:18 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:18.958+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:19.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:19.205+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:19.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:19.205+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:19.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:19.205+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:19.337 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:19.206+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:19.681 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:19 smithi102 podman[72718]: 2024-03-09 00:19:19.414495168 +0000 UTC m=+5.927450309 container cleanup e0cb21bb69f5b1ec602d280106c3c10c33df7fa3dcef72da146707411e915f71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-09T00:19:19.681 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:19 smithi102 bash[72718]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7 2024-03-09T00:19:19.681 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:19.457+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:19.681 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:19.457+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:19.682 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:19.457+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:19.682 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:19.457+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:19.682 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:19 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:19.457+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:20.013 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:19 smithi102 podman[72734]: 2024-03-09 00:19:19.682260787 +0000 UTC m=+0.636332223 container remove e0cb21bb69f5b1ec602d280106c3c10c33df7fa3dcef72da146707411e915f71 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-09T00:19:20.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:19.692+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:20.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:19.692+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:20.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:19.692+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:20.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:19.692+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:20.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:19 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:19.692+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:20.247 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:19.994+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:20.247 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:19.994+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:20.247 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:19.994+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:20.247 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:19 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:19.994+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:20.276 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:19.922637584 +0000 UTC m=+0.017442507 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:20.276 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:20.024822806 +0000 UTC m=+0.119627715 container create 193a8bb7a602085500d05f6628ea15a0c937932e32a0d8acc93d1d9bb3cd4b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:19:20.276 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:20.277564946 +0000 UTC m=+0.372369858 container init 193a8bb7a602085500d05f6628ea15a0c937932e32a0d8acc93d1d9bb3cd4b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7-deactivate, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-09T00:19:20.576 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:20.423+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:20.576 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:20.423+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:20.423+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:20.423+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:20 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:20.423+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:20.280807592 +0000 UTC m=+0.375612499 container start 193a8bb7a602085500d05f6628ea15a0c937932e32a0d8acc93d1d9bb3cd4b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:20.419517984 +0000 UTC m=+0.514322929 container attach 193a8bb7a602085500d05f6628ea15a0c937932e32a0d8acc93d1d9bb3cd4b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7-deactivate, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-09T00:19:20.577 INFO:journalctl@ceph.osd.7.smithi102.stdout:Mar 09 00:19:20 smithi102 podman[72826]: 2024-03-09 00:19:20.577652469 +0000 UTC m=+0.672457382 container died 193a8bb7a602085500d05f6628ea15a0c937932e32a0d8acc93d1d9bb3cd4b56 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-7-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-09T00:19:20.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:20.247+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:20.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:20.247+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:20.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:20.247+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:20.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:20 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:20.247+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:21.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:20.672+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:21.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:20.672+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:21.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:20.672+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:21.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:20.672+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:21.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:20 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:20.672+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:21.196 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.7.service' 2024-03-09T00:19:21.239 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:21.240 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-09T00:19:21.240 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-09T00:19:21.240 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.8 2024-03-09T00:19:21.281 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.026+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:21.281 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.026+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:21.281 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.026+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:21.282 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.026+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:21.450 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:21.391+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:21.450 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:21.391+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:21.450 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:21.391+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:21.450 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:21.391+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:21.450 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:21 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:21.391+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:21.554 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.282+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:21.555 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.282+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:21.555 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.282+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:21.555 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.282+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:21.555 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 systemd[1]: Stopping Ceph osd.8 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:21.555 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.555+0000 7f55ab382700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:21.556 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.555+0000 7f55ab382700 -1 osd.8 109 *** Got signal Terminated *** 2024-03-09T00:19:21.556 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:21.555+0000 7f55ab382700 -1 osd.8 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:22.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:21.678+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:22.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:21.678+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:22.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:21.678+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:22.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:21.678+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:22.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:21 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:21.678+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:22.241 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:22.242 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:22.242 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:22.242 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:21.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:22.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:22.242+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:22.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:22.242+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:22.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:22.242+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:22.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:22.242+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:22.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:22.345+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:22.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:22.345+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:22.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:22.345+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:22.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:22.345+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:22.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:22 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:22.345+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:23.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:22.728+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:23.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:22.728+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:23.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:22.728+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:23.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:22.728+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:23.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:22 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:22.728+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:23.250 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:22.972+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:23.250 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:22.972+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:23.250 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:22.972+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:23.250 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:22 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:22.972+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:23.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:23.250+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:23.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:23.250+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:23.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:23.250+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:23.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:23.250+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:23.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:23.386+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:23.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:23.386+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:23.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:23.386+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:23.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:23.386+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:23.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:23 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:23.386+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:24.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:23.760+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:24.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:23.760+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:24.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:23.760+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:24.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:23.760+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:24.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:23 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:23.760+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:24.288 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.006+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:24.288 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.006+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:24.288 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.006+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:24.288 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.006+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:24.562 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:24.288+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:24.562 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:24.288+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:24.562 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:24.288+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:24.562 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:24.288+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:24.665 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:24.414+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:24.665 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:24.414+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:24.665 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:24.414+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:24.665 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:24.414+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:24.665 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:24 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:24.414+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:25.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:25.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:25.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:25.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:25.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:25.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:24 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:24.792+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:25.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.973+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:25.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.973+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:25.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.973+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:25.328 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:24 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:24.973+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:25.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:25.328+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:25.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:25.328+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:25.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:25.328+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:25.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:25.328+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:25.587 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:25.328+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.270422+0000 front 2024-03-09T00:19:00.270445+0000 (oldest deadline 2024-03-09T00:19:24.370654+0000) 2024-03-09T00:19:25.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:25.456+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:25.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:25.456+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:25.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:25.456+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:25.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:25.456+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:25.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:25 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:25.456+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:26.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:26.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:26.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:26.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:26.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:26.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:25 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:25.786+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:26.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:25.959+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:26.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:25.959+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:26.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:25.959+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:26.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:25 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:25.959+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:26.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:26.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:26.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:26.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:26.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:26.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:26 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:26.502+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:26.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:26.343+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.167158+0000 front 2024-03-09T00:18:29.167174+0000 (oldest deadline 2024-03-09T00:18:55.067153+0000) 2024-03-09T00:19:26.836 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:26.343+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.067613+0000 front 2024-03-09T00:18:35.067596+0000 (oldest deadline 2024-03-09T00:19:00.967774+0000) 2024-03-09T00:19:26.837 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:26.343+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:42.668551+0000 front 2024-03-09T00:18:42.668577+0000 (oldest deadline 2024-03-09T00:19:07.368657+0000) 2024-03-09T00:19:26.837 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:26.343+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.469736+0000 front 2024-03-09T00:18:51.469755+0000 (oldest deadline 2024-03-09T00:19:16.169643+0000) 2024-03-09T00:19:26.837 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8[55117]: 2024-03-09T00:19:26.343+0000 7f55a7976700 -1 osd.8 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.270422+0000 front 2024-03-09T00:19:00.270445+0000 (oldest deadline 2024-03-09T00:19:24.370654+0000) 2024-03-09T00:19:27.155 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:26 smithi179 podman[73916]: 2024-03-09 00:19:26.875376149 +0000 UTC m=+5.445837819 container died 9576566eae8662093eed54a70422beb985f592413ff537baee38eefba52e284b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-09T00:19:27.155 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:26.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:27.155 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:26.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:27.155 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:26.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:27.156 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:26.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:27.156 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:26 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:26.978+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:27.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:27.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:27.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:27.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:27.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:27.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:26 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:26.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:27.488 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:27 smithi179 podman[73916]: 2024-03-09 00:19:27.254629229 +0000 UTC m=+5.825090889 container cleanup 9576566eae8662093eed54a70422beb985f592413ff537baee38eefba52e284b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True) 2024-03-09T00:19:27.489 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:27 smithi179 bash[73916]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8 2024-03-09T00:19:27.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:27.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:27.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:27.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:27.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:27.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:27 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:27.474+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:27.774 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:27 smithi179 podman[73945]: 2024-03-09 00:19:27.489397728 +0000 UTC m=+0.612141317 container remove 9576566eae8662093eed54a70422beb985f592413ff537baee38eefba52e284b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0) 2024-03-09T00:19:28.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:27 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:27.988+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:28.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:27 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:27.988+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:28.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:27 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:27.988+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:28.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:27 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:27.988+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:28.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:27 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:27.988+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:28.088 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:27 smithi179 podman[74034]: 2024-03-09 00:19:27.678722521 +0000 UTC m=+0.018204726 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:28.088 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:27 smithi179 podman[74034]: 2024-03-09 00:19:27.790087214 +0000 UTC m=+0.129569405 container create f2b6f0e765b9888666b8be347a843520b0e28ee66b95159f3d26841fcee84707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8-deactivate, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-09T00:19:28.088 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:28 smithi179 podman[74034]: 2024-03-09 00:19:28.017726106 +0000 UTC m=+0.357208308 container init f2b6f0e765b9888666b8be347a843520b0e28ee66b95159f3d26841fcee84707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-09T00:19:28.088 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:28 smithi179 podman[74034]: 2024-03-09 00:19:28.020761986 +0000 UTC m=+0.360244179 container start f2b6f0e765b9888666b8be347a843520b0e28ee66b95159f3d26841fcee84707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8-deactivate, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-09T00:19:28.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:28.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:28.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:28.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:28.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:28.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:27 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:27.780+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:28.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:28 smithi179 podman[74034]: 2024-03-09 00:19:28.117698852 +0000 UTC m=+0.457181060 container attach f2b6f0e765b9888666b8be347a843520b0e28ee66b95159f3d26841fcee84707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=) 2024-03-09T00:19:28.586 INFO:journalctl@ceph.osd.8.smithi179.stdout:Mar 09 00:19:28 smithi179 podman[74034]: 2024-03-09 00:19:28.313527719 +0000 UTC m=+0.653009903 container died f2b6f0e765b9888666b8be347a843520b0e28ee66b95159f3d26841fcee84707 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-8-deactivate, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, io.buildah.version=1.33.5) 2024-03-09T00:19:28.944 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.8.service' 2024-03-09T00:19:29.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:29.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:29.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:29.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:29.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:29.015 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:28 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:28.514+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:29.030 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:29.033 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-09T00:19:29.034 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-09T00:19:29.034 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.9 2024-03-09T00:19:29.063 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:29.064 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:29.064 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:29.064 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:29.064 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:29.064 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:28 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:28.742+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:29.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:28.975+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:29.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:28.975+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:29.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:28.975+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:29.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:28.975+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:29.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:28.975+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:29.360 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 systemd[1]: Stopping Ceph osd.9 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:29.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.360+0000 7f0baee21700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:29.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.360+0000 7f0baee21700 -1 osd.9 109 *** Got signal Terminated *** 2024-03-09T00:19:29.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.360+0000 7f0baee21700 -1 osd.9 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:30.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:29 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:29.539+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:30.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:30.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:30.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:30.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:30.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:30.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:29 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:29.774+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:30.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:29 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:29.929+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:30.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:29 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:29.929+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:30.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:29 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:29.929+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:30.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:29 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:29.929+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:30.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:29 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:29.929+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:31.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:30 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:30.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:31.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:31.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:31.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:31.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:31.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:31.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:30 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:30.758+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:31.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:30 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:30.881+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:31.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:30 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:30.881+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:31.336 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:30 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:30.881+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:31.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:30 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:30.881+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:31.337 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:30 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:30.881+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:31.874 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:31.874 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:31.874 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:31.875 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:31.875 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:31.875 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:31 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:31.534+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:32.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:31 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:31.835+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:32.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:31 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:31.835+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:32.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:31 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:31.835+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:32.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:31 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:31.835+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:32.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:31 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:31.835+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:32.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.805+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:32.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.806+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:32.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.806+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:32.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.806+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:32.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.806+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:32.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:31 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:31.806+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:32.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:32 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:32.570+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:33.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:32 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:32.790+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:33.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:32 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:32.790+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:33.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:32 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:32.790+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:33.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:32 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:32.790+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:33.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:32 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:32.790+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:33.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:33.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:33.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:33.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:33.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:33.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:32 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:32.841+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:34.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:33 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:33.532+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:34.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:34.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:34.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:34.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:34.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:34.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:33.807+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:34.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:21.861620+0000 front 2024-03-09T00:18:21.861648+0000 (oldest deadline 2024-03-09T00:18:47.161837+0000) 2024-03-09T00:19:34.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:27.162363+0000 front 2024-03-09T00:18:27.162480+0000 (oldest deadline 2024-03-09T00:18:51.862387+0000) 2024-03-09T00:19:34.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:37.664012+0000 front 2024-03-09T00:18:37.663926+0000 (oldest deadline 2024-03-09T00:18:59.963983+0000) 2024-03-09T00:19:34.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:44.565485+0000 front 2024-03-09T00:18:44.565546+0000 (oldest deadline 2024-03-09T00:19:09.865595+0000) 2024-03-09T00:19:34.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.966706+0000 front 2024-03-09T00:18:53.966886+0000 (oldest deadline 2024-03-09T00:19:16.866726+0000) 2024-03-09T00:19:34.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:33 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9[67559]: 2024-03-09T00:19:33.869+0000 7f0ba740d700 -1 osd.9 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.368045+0000 front 2024-03-09T00:19:00.367958+0000 (oldest deadline 2024-03-09T00:19:24.467832+0000) 2024-03-09T00:19:34.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:34.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:34 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:34.483+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:35.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:35.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:35.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:35.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:35.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:35.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:34 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:34.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:35.135 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:34 smithi073 podman[106824]: 2024-03-09 00:19:34.795465435 +0000 UTC m=+5.566160110 container died f305c848a3c6e972db6b22abc9c9f1d185b98cc51d208fc74b460ce6652735aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-09T00:19:35.406 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:35 smithi073 podman[106824]: 2024-03-09 00:19:35.218486999 +0000 UTC m=+5.989181672 container cleanup f305c848a3c6e972db6b22abc9c9f1d185b98cc51d208fc74b460ce6652735aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-09T00:19:35.406 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:35 smithi073 bash[106824]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9 2024-03-09T00:19:35.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:35.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:35.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:35.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:35.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:35.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:35.767 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:35 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:35.460+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:35.873 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:35 smithi073 podman[106841]: 2024-03-09 00:19:35.45770201 +0000 UTC m=+0.652750705 container remove f305c848a3c6e972db6b22abc9c9f1d185b98cc51d208fc74b460ce6652735aa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:36.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:35 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:35.716+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:36.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:35 smithi073 podman[106931]: 2024-03-09 00:19:35.777599487 +0000 UTC m=+0.017824850 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:36.156 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:35 smithi073 podman[106931]: 2024-03-09 00:19:35.883535009 +0000 UTC m=+0.123760410 container create 6072c0fc65aff77eae02e826ce4c693be1cac0462992027270612c0b82ffe536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux ) 2024-03-09T00:19:36.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:36 smithi073 podman[106931]: 2024-03-09 00:19:36.110820036 +0000 UTC m=+0.351045404 container init 6072c0fc65aff77eae02e826ce4c693be1cac0462992027270612c0b82ffe536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-09T00:19:36.157 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:36 smithi073 podman[106931]: 2024-03-09 00:19:36.11390881 +0000 UTC m=+0.354134171 container start 6072c0fc65aff77eae02e826ce4c693be1cac0462992027270612c0b82ffe536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.build-date=20240307) 2024-03-09T00:19:36.656 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:36 smithi073 podman[106931]: 2024-03-09 00:19:36.202508259 +0000 UTC m=+0.442733629 container attach 6072c0fc65aff77eae02e826ce4c693be1cac0462992027270612c0b82ffe536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482) 2024-03-09T00:19:36.657 INFO:journalctl@ceph.osd.9.smithi073.stdout:Mar 09 00:19:36 smithi073 podman[106931]: 2024-03-09 00:19:36.398911204 +0000 UTC m=+0.639136570 container died 6072c0fc65aff77eae02e826ce4c693be1cac0462992027270612c0b82ffe536 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-9-deactivate, org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-09T00:19:36.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:36.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:36.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:36.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:36.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:36.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:36.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:36 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:36.425+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:36.973 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.9.service' 2024-03-09T00:19:37.059 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:37.059 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-09T00:19:37.059 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-09T00:19:37.059 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.10 2024-03-09T00:19:37.088 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:37.088 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:37.089 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:37.089 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:37.089 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:37.089 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:36 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:36.759+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:37.375 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 systemd[1]: Stopping Ceph osd.10 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:37.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.375+0000 7f78945c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:37.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.375+0000 7f78945c3700 -1 osd.10 109 *** Got signal Terminated *** 2024-03-09T00:19:37.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.375+0000 7f78945c3700 -1 osd.10 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:37.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:37 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:37.458+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:38.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:38.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:38.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:38.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:38.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:38.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:37 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:37.760+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:38.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:38.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:38.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:38.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:38.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:38.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:38.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:38 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:38.495+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:39.068 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.770+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:39.068 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.770+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:39.068 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.770+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:39.068 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.771+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:39.069 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.771+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:39.069 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:38 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:38.771+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:39.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:39.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:39 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:39.501+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:40.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:40.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:40.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:40.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:40.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:40.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:39 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:39.762+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:40.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:40.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:40.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:40.765 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:40.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:40.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:40.766 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:40 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:40.454+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:41.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:41.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:40 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:40.745+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:41.763 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.015950+0000 front 2024-03-09T00:18:20.015964+0000 (oldest deadline 2024-03-09T00:18:45.916006+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:29.317541+0000 front 2024-03-09T00:18:29.317570+0000 (oldest deadline 2024-03-09T00:18:51.017736+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:36.120039+0000 front 2024-03-09T00:18:36.120170+0000 (oldest deadline 2024-03-09T00:19:00.819948+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6814 osd.3 since back 2024-03-09T00:18:45.421644+0000 front 2024-03-09T00:18:45.421724+0000 (oldest deadline 2024-03-09T00:19:09.422177+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:51.023239+0000 front 2024-03-09T00:18:51.023296+0000 (oldest deadline 2024-03-09T00:19:15.723469+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:00.325135+0000 front 2024-03-09T00:19:00.325094+0000 (oldest deadline 2024-03-09T00:19:25.625143+0000) 2024-03-09T00:19:41.764 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:41 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10[60797]: 2024-03-09T00:19:41.432+0000 7f788cbaf700 -1 osd.10 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:10.226515+0000 front 2024-03-09T00:19:10.226770+0000 (oldest deadline 2024-03-09T00:19:33.726169+0000) 2024-03-09T00:19:42.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:42.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:42.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:42.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:42.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:42.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:42.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:41 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:41.767+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:43.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:43.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:42 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:42.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:43.109 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:42 smithi102 podman[72944]: 2024-03-09 00:19:42.79909691 +0000 UTC m=+5.542816264 container died e6f06c4ad7e925f333ecb4223a9cba3da49c63bdf09d92690e203896f145128c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, GIT_BRANCH=HEAD, ceph=True) 2024-03-09T00:19:43.109 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[72944]: 2024-03-09 00:19:43.110272127 +0000 UTC m=+5.853991481 container cleanup e6f06c4ad7e925f333ecb4223a9cba3da49c63bdf09d92690e203896f145128c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-09T00:19:43.513 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 bash[72944]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10 2024-03-09T00:19:43.514 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[72962]: 2024-03-09 00:19:43.311964644 +0000 UTC m=+0.513914098 container remove e6f06c4ad7e925f333ecb4223a9cba3da49c63bdf09d92690e203896f145128c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-09T00:19:44.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[73053]: 2024-03-09 00:19:43.622931616 +0000 UTC m=+0.018243581 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:44.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[73053]: 2024-03-09 00:19:43.730321109 +0000 UTC m=+0.125633057 container create f44c1e164d1a7bc6c9b1d5c6cef6a39fdaab20556526ac524fd1276e770b7d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-09T00:19:44.014 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[73053]: 2024-03-09 00:19:43.933724104 +0000 UTC m=+0.329036071 container init f44c1e164d1a7bc6c9b1d5c6cef6a39fdaab20556526ac524fd1276e770b7d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-09T00:19:44.015 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:43 smithi102 podman[73053]: 2024-03-09 00:19:43.937138373 +0000 UTC m=+0.332450336 container start f44c1e164d1a7bc6c9b1d5c6cef6a39fdaab20556526ac524fd1276e770b7d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:19:44.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:44.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:43 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:43.679+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:44.510 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:44 smithi102 podman[73053]: 2024-03-09 00:19:44.025004439 +0000 UTC m=+0.420316397 container attach f44c1e164d1a7bc6c9b1d5c6cef6a39fdaab20556526ac524fd1276e770b7d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-09T00:19:44.511 INFO:journalctl@ceph.osd.10.smithi102.stdout:Mar 09 00:19:44 smithi102 podman[73053]: 2024-03-09 00:19:44.228132237 +0000 UTC m=+0.623444192 container died f44c1e164d1a7bc6c9b1d5c6cef6a39fdaab20556526ac524fd1276e770b7d1d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, ceph=True) 2024-03-09T00:19:44.829 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.10.service' 2024-03-09T00:19:44.873 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:44.874 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-09T00:19:44.874 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-09T00:19:44.874 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.11 2024-03-09T00:19:45.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:44.717+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:45.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:44 smithi179 systemd[1]: Stopping Ceph osd.11 for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:45.586 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.172+0000 7fb79b89f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:45.587 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.172+0000 7fb79b89f700 -1 osd.11 109 *** Got signal Terminated *** 2024-03-09T00:19:45.587 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.172+0000 7fb79b89f700 -1 osd.11 109 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:46.095 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:46.096 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:45 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:45.706+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:47.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:47.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:46 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:46.698+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:48.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:48.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:47 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:47.695+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-03-09T00:19:24.067101+0000 front 2024-03-09T00:19:24.067065+0000 (oldest deadline 2024-03-09T00:19:46.967354+0000) 2024-03-09T00:19:49.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:49.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:49.088 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:48 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:48.718+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-03-09T00:19:24.067101+0000 front 2024-03-09T00:19:24.067065+0000 (oldest deadline 2024-03-09T00:19:46.967354+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6806 osd.0 since back 2024-03-09T00:18:20.357302+0000 front 2024-03-09T00:18:20.357246+0000 (oldest deadline 2024-03-09T00:18:45.057205+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6804 osd.1 since back 2024-03-09T00:18:30.258710+0000 front 2024-03-09T00:18:30.258754+0000 (oldest deadline 2024-03-09T00:18:55.558863+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6804 osd.2 since back 2024-03-09T00:18:35.559402+0000 front 2024-03-09T00:18:35.559269+0000 (oldest deadline 2024-03-09T00:19:00.259391+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6812 osd.4 since back 2024-03-09T00:18:53.161811+0000 front 2024-03-09T00:18:53.161795+0000 (oldest deadline 2024-03-09T00:19:19.061709+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6812 osd.5 since back 2024-03-09T00:19:01.962786+0000 front 2024-03-09T00:19:01.962873+0000 (oldest deadline 2024-03-09T00:19:26.662893+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.73:6822 osd.6 since back 2024-03-09T00:19:09.464871+0000 front 2024-03-09T00:19:09.464895+0000 (oldest deadline 2024-03-09T00:19:33.564876+0000) 2024-03-09T00:19:50.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.102:6820 osd.7 since back 2024-03-09T00:19:14.666138+0000 front 2024-03-09T00:19:14.666095+0000 (oldest deadline 2024-03-09T00:19:39.966057+0000) 2024-03-09T00:19:50.088 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:49 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11[62121]: 2024-03-09T00:19:49.750+0000 7fb793e8b700 -1 osd.11 109 heartbeat_check: no reply from 172.21.15.179:6820 osd.8 since back 2024-03-09T00:19:24.067101+0000 front 2024-03-09T00:19:24.067065+0000 (oldest deadline 2024-03-09T00:19:46.967354+0000) 2024-03-09T00:19:50.797 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:50 smithi179 podman[74139]: 2024-03-09 00:19:50.49775638 +0000 UTC m=+5.427143014 container died 8c614f05dd61628f4f782482e82ca061ba022869be48976247a961ea39e9d198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0) 2024-03-09T00:19:50.798 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:50 smithi179 podman[74139]: 2024-03-09 00:19:50.797972116 +0000 UTC m=+5.727358751 container cleanup 8c614f05dd61628f4f782482e82ca061ba022869be48976247a961ea39e9d198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:19:51.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:50 smithi179 bash[74139]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11 2024-03-09T00:19:51.087 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74156]: 2024-03-09 00:19:51.015659002 +0000 UTC m=+0.518249480 container remove 8c614f05dd61628f4f782482e82ca061ba022869be48976247a961ea39e9d198 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:19:51.586 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.216001761 +0000 UTC m=+0.018158593 image pull a272146cb810b1fd4a53627f019472132e30966d5c25b01bfc44a001295f1249 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd 2024-03-09T00:19:51.587 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.316321078 +0000 UTC m=+0.118477884 container create d2f03304597f6262138041bf45f35f09787ea958646cfa5784fa0ed4b6f84c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS) 2024-03-09T00:19:51.587 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.552254868 +0000 UTC m=+0.354411664 container init d2f03304597f6262138041bf45f35f09787ea958646cfa5784fa0ed4b6f84c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-09T00:19:51.587 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.555286988 +0000 UTC m=+0.357443789 container start d2f03304597f6262138041bf45f35f09787ea958646cfa5784fa0ed4b6f84c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_BRANCH=HEAD) 2024-03-09T00:19:52.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.635532693 +0000 UTC m=+0.437689512 container attach d2f03304597f6262138041bf45f35f09787ea958646cfa5784fa0ed4b6f84c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11-deactivate, ceph=True, RELEASE=rishabh-2024mar8-7d63482, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307) 2024-03-09T00:19:52.086 INFO:journalctl@ceph.osd.11.smithi179.stdout:Mar 09 00:19:51 smithi179 podman[74250]: 2024-03-09 00:19:51.840070766 +0000 UTC m=+0.642227585 container died d2f03304597f6262138041bf45f35f09787ea958646cfa5784fa0ed4b6f84c62 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-osd-11-deactivate, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-09T00:19:52.379 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@osd.11.service' 2024-03-09T00:19:52.422 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:52.422 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-09T00:19:52.423 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-09T00:19:52.423 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.a 2024-03-09T00:19:52.782 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:52 smithi073 systemd[1]: Stopping Ceph mds.a for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:53.157 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:52 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a[74586]: 2024-03-09T00:19:52.781+0000 7fdb0d688700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:53.158 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:52 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a[74586]: 2024-03-09T00:19:52.781+0000 7fdb0d688700 -1 mds.a *** got signal Terminated *** 2024-03-09T00:19:54.274 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:53 smithi073 podman[107053]: 2024-03-09 00:19:53.947436731 +0000 UTC m=+1.325079161 container died 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0) 2024-03-09T00:19:54.573 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:54 smithi073 podman[107053]: 2024-03-09 00:19:54.357421462 +0000 UTC m=+1.735063897 container cleanup 7b33c054aa0e4ccdbb51ec7275d0f14d05c9ebb8c7c2cab7200629490fd99c8e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2) 2024-03-09T00:19:54.574 INFO:journalctl@ceph.mds.a.smithi073.stdout:Mar 09 00:19:54 smithi073 bash[107053]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-a 2024-03-09T00:19:54.693 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.a.service' 2024-03-09T00:19:54.732 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:54.733 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-09T00:19:54.733 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-09T00:19:54.733 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.d 2024-03-09T00:19:54.906 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:54 smithi073 systemd[1]: Stopping Ceph mds.d for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:55.406 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:55 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d[73606]: 2024-03-09T00:19:55.060+0000 7f3510da5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.d -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:55.406 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:55 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d[73606]: 2024-03-09T00:19:55.060+0000 7f3510da5700 -1 mds.d *** got signal Terminated *** 2024-03-09T00:19:58.120 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:57 smithi073 podman[107174]: 2024-03-09 00:19:57.84822139 +0000 UTC m=+2.893843275 container died c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-09T00:19:58.406 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:58 smithi073 podman[107174]: 2024-03-09 00:19:58.202661192 +0000 UTC m=+3.248283077 container cleanup c3660eafc5f4b2a0aca7cfe55011874de7cb08791f84904078c611bf33143aee (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-09T00:19:58.406 INFO:journalctl@ceph.mds.d.smithi073.stdout:Mar 09 00:19:58 smithi073 bash[107174]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-d 2024-03-09T00:19:58.627 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.d.service' 2024-03-09T00:19:58.663 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:19:58.663 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-09T00:19:58.664 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-09T00:19:58.664 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.g 2024-03-09T00:19:58.989 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:19:58 smithi073 systemd[1]: Stopping Ceph mds.g for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:19:59.406 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:19:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g[74217]: 2024-03-09T00:19:58.989+0000 7f98738c3700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.g -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:19:59.406 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:19:58 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g[74217]: 2024-03-09T00:19:58.989+0000 7f98738c3700 -1 mds.g *** got signal Terminated *** 2024-03-09T00:20:03.364 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:20:03 smithi073 podman[107299]: 2024-03-09 00:20:03.001346755 +0000 UTC m=+4.120153054 container died aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:20:03.648 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:20:03 smithi073 podman[107299]: 2024-03-09 00:20:03.447903297 +0000 UTC m=+4.566709596 container cleanup aa805f636c59ace2242d1808a6670075489d93bb3bbff4e1a02cd60ed09ff529 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=rishabh-2024mar8-7d63482, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:20:03.648 INFO:journalctl@ceph.mds.g.smithi073.stdout:Mar 09 00:20:03 smithi073 bash[107299]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-g 2024-03-09T00:20:03.867 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.g.service' 2024-03-09T00:20:03.903 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:03.903 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-09T00:20:03.904 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-09T00:20:03.904 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.j 2024-03-09T00:20:04.156 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:03 smithi073 systemd[1]: Stopping Ceph mds.j for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:04.656 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j[73258]: 2024-03-09T00:20:04.243+0000 7f16fed49700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.j -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:04.656 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:04 smithi073 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j[73258]: 2024-03-09T00:20:04.243+0000 7f16fed49700 -1 mds.j *** got signal Terminated *** 2024-03-09T00:20:07.168 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:06 smithi073 podman[107421]: 2024-03-09 00:20:06.918301908 +0000 UTC m=+2.792136116 container died 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-09T00:20:07.526 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:07 smithi073 podman[107421]: 2024-03-09 00:20:07.27704282 +0000 UTC m=+3.150877025 container cleanup 070b7a4d5e5dbaf77fcd366dbedf2cf3400d73ac8fb506ebc7181737f3fa3b30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, ceph=True) 2024-03-09T00:20:07.527 INFO:journalctl@ceph.mds.j.smithi073.stdout:Mar 09 00:20:07 smithi073 bash[107421]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-j 2024-03-09T00:20:07.689 DEBUG:teuthology.orchestra.run.smithi073:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.j.service' 2024-03-09T00:20:07.723 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:07.723 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-09T00:20:07.724 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-09T00:20:07.724 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.b 2024-03-09T00:20:08.014 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:07 smithi102 systemd[1]: Stopping Ceph mds.b for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:08.514 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: 2024-03-09T00:20:08.038+0000 7efc9d599700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:08.514 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:08 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b[66497]: 2024-03-09T00:20:08.038+0000 7efc9d599700 -1 mds.b *** got signal Terminated *** 2024-03-09T00:20:09.729 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:09 smithi102 podman[73161]: 2024-03-09 00:20:09.455343392 +0000 UTC m=+1.536173239 container died bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240307, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-09T00:20:10.014 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:09 smithi102 podman[73161]: 2024-03-09 00:20:09.846170732 +0000 UTC m=+1.927000589 container cleanup bfe0134cfd31fd534fbe8a83e08bd50dff0ac2cc5c4ccff492827fa996ac306d (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , ceph=True) 2024-03-09T00:20:10.014 INFO:journalctl@ceph.mds.b.smithi102.stdout:Mar 09 00:20:09 smithi102 bash[73161]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-b 2024-03-09T00:20:10.271 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.b.service' 2024-03-09T00:20:10.309 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:10.310 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-09T00:20:10.310 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-09T00:20:10.310 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.e 2024-03-09T00:20:10.624 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:10 smithi102 systemd[1]: Stopping Ceph mds.e for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:11.014 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e[65880]: 2024-03-09T00:20:10.624+0000 7f11a79f8700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:11.014 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:10 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e[65880]: 2024-03-09T00:20:10.624+0000 7f11a79f8700 -1 mds.e *** got signal Terminated *** 2024-03-09T00:20:13.092 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:12 smithi102 podman[73293]: 2024-03-09 00:20:12.786495147 +0000 UTC m=+2.259393574 container died fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=rishabh-2024mar8-7d63482, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-09T00:20:13.426 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:13 smithi102 podman[73293]: 2024-03-09 00:20:13.167530052 +0000 UTC m=+2.640428482 container cleanup fde9e6bd001c1321313c3c6f144f49eef3e55193133a376da070c6f53e71a854 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e, GIT_CLEAN=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:20:13.427 INFO:journalctl@ceph.mds.e.smithi102.stdout:Mar 09 00:20:13 smithi102 bash[73293]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-e 2024-03-09T00:20:13.589 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.e.service' 2024-03-09T00:20:13.625 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:13.625 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-09T00:20:13.626 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-09T00:20:13.626 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.h 2024-03-09T00:20:13.955 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:13 smithi102 systemd[1]: Stopping Ceph mds.h for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:14.264 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h[66188]: 2024-03-09T00:20:13.955+0000 7f2558e29700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.h -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:14.264 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:13 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h[66188]: 2024-03-09T00:20:13.955+0000 7f2558e29700 -1 mds.h *** got signal Terminated *** 2024-03-09T00:20:17.688 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:17 smithi102 podman[73414]: 2024-03-09 00:20:17.40940206 +0000 UTC m=+3.561732813 container died 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.build-date=20240307, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-09T00:20:18.014 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:17 smithi102 podman[73414]: 2024-03-09 00:20:17.780684921 +0000 UTC m=+3.933015674 container cleanup 937a46e83c3a1bf695fb60300b9ab001af4136f6312779f866aa22d6260cc155 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240307, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:20:18.014 INFO:journalctl@ceph.mds.h.smithi102.stdout:Mar 09 00:20:17 smithi102 bash[73414]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-h 2024-03-09T00:20:18.155 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.h.service' 2024-03-09T00:20:18.190 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:18.190 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-09T00:20:18.190 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-09T00:20:18.191 DEBUG:teuthology.orchestra.run.smithi102:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.k 2024-03-09T00:20:18.764 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:18 smithi102 systemd[1]: Stopping Ceph mds.k for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:18.764 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k[65554]: 2024-03-09T00:20:18.517+0000 7f43822a4700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:18.764 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:18 smithi102 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k[65554]: 2024-03-09T00:20:18.517+0000 7f43822a4700 -1 mds.k *** got signal Terminated *** 2024-03-09T00:20:20.184 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:19 smithi102 podman[73534]: 2024-03-09 00:20:19.926196409 +0000 UTC m=+1.519968407 container died 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k, org.label-schema.build-date=20240307, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-09T00:20:20.514 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:20 smithi102 podman[73534]: 2024-03-09 00:20:20.284622835 +0000 UTC m=+1.878394834 container cleanup 492fb651b03c16f73835b87ac1cbaef9428e2fc3cb28077e155c1f37d9831bce (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, RELEASE=rishabh-2024mar8-7d63482, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-09T00:20:20.514 INFO:journalctl@ceph.mds.k.smithi102.stdout:Mar 09 00:20:20 smithi102 bash[73534]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-k 2024-03-09T00:20:20.704 DEBUG:teuthology.orchestra.run.smithi102:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.k.service' 2024-03-09T00:20:20.737 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:20.738 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-09T00:20:20.738 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-09T00:20:20.738 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.c 2024-03-09T00:20:21.040 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:20 smithi179 systemd[1]: Stopping Ceph mds.c for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:21.336 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c[67285]: 2024-03-09T00:20:21.040+0000 7fa64efae700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:21.336 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:21 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c[67285]: 2024-03-09T00:20:21.040+0000 7fa64efae700 -1 mds.c *** got signal Terminated *** 2024-03-09T00:20:22.715 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:22 smithi179 podman[74367]: 2024-03-09 00:20:22.428911066 +0000 UTC m=+1.493813813 container died 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-09T00:20:23.016 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:22 smithi179 podman[74367]: 2024-03-09 00:20:22.807242787 +0000 UTC m=+1.872145537 container cleanup 6dbee69e59449113625432bfdfd0295c79f9b7140cdf4a40ca69cb4004af807f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-09T00:20:23.016 INFO:journalctl@ceph.mds.c.smithi179.stdout:Mar 09 00:20:22 smithi179 bash[74367]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-c 2024-03-09T00:20:23.133 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.c.service' 2024-03-09T00:20:23.171 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:23.172 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-09T00:20:23.172 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-09T00:20:23.172 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.f 2024-03-09T00:20:23.336 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:23 smithi179 systemd[1]: Stopping Ceph mds.f for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:23.836 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f[66979]: 2024-03-09T00:20:23.485+0000 7f5a5d97c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.f -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:23.837 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:23 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f[66979]: 2024-03-09T00:20:23.485+0000 7f5a5d97c700 -1 mds.f *** got signal Terminated *** 2024-03-09T00:20:27.750 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:27 smithi179 podman[74490]: 2024-03-09 00:20:27.492814302 +0000 UTC m=+4.100495487 container died 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240307) 2024-03-09T00:20:27.751 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:27 smithi179 podman[74490]: 2024-03-09 00:20:27.735480104 +0000 UTC m=+4.343161277 container cleanup 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240307, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-09T00:20:27.751 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:27 smithi179 bash[74490]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f 2024-03-09T00:20:28.153 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.f.service' 2024-03-09T00:20:28.177 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:27 smithi179 podman[74504]: 2024-03-09 00:20:27.911756343 +0000 UTC m=+0.414616898 container remove 0de693ebc82ce485883ccff9407e47b835a1645487a3e55759a7a286e797ff2c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-f, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240307, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-09T00:20:28.177 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:28 smithi179 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.f.service: Deactivated successfully. 2024-03-09T00:20:28.178 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:28 smithi179 systemd[1]: Stopped Ceph mds.f for 44f886a8-dda7-11ee-95c6-87774f69a715. 2024-03-09T00:20:28.178 INFO:journalctl@ceph.mds.f.smithi179.stdout:Mar 09 00:20:28 smithi179 systemd[1]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.f.service: Consumed 1.102s CPU time. 2024-03-09T00:20:28.189 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:28.189 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-09T00:20:28.189 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-09T00:20:28.189 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.i 2024-03-09T00:20:28.531 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:28 smithi179 systemd[1]: Stopping Ceph mds.i for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:28.837 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: 2024-03-09T00:20:28.530+0000 7ffb39f4e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:28.837 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:28 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i[67591]: 2024-03-09T00:20:28.530+0000 7ffb39f4e700 -1 mds.i *** got signal Terminated *** 2024-03-09T00:20:32.837 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:32 smithi179 podman[74612]: 2024-03-09 00:20:32.535190399 +0000 UTC m=+4.111905065 container died 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=rishabh-2024mar8-7d63482, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240307, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-09T00:20:33.256 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:32 smithi179 podman[74612]: 2024-03-09 00:20:32.997032028 +0000 UTC m=+4.573746687 container cleanup 643e59857f217595846b4c7bbd782188cbe5b08285b950990c308e52da595fde (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240307, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=rishabh-2024mar8-7d63482, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-09T00:20:33.256 INFO:journalctl@ceph.mds.i.smithi179.stdout:Mar 09 00:20:32 smithi179 bash[74612]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-i 2024-03-09T00:20:33.418 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.i.service' 2024-03-09T00:20:33.452 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:33.452 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-09T00:20:33.453 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-09T00:20:33.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.l 2024-03-09T00:20:33.766 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:33 smithi179 systemd[1]: Stopping Ceph mds.l for 44f886a8-dda7-11ee-95c6-87774f69a715... 2024-03-09T00:20:34.087 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l[66658]: 2024-03-09T00:20:33.766+0000 7f55e3765700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.l -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-09T00:20:34.087 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:33 smithi179 ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l[66658]: 2024-03-09T00:20:33.766+0000 7f55e3765700 -1 mds.l *** got signal Terminated *** 2024-03-09T00:20:37.275 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:37 smithi179 podman[74732]: 2024-03-09 00:20:37.006354387 +0000 UTC m=+3.342423886 container died 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=rishabh-2024mar8-7d63482, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240307, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-09T00:20:37.587 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:37 smithi179 podman[74732]: 2024-03-09 00:20:37.384011785 +0000 UTC m=+3.720081282 container cleanup 8220a0c5a91f4675cb72a21ea931a1bda59afe175f82b9057577201cf711a125 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:70b1c28ffbd1be392f87b7c6c0373971a00211c0624517727a0cab908539babd, name=ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l, org.label-schema.build-date=20240307, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=rishabh-2024mar8-7d63482, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-09T00:20:37.587 INFO:journalctl@ceph.mds.l.smithi179.stdout:Mar 09 00:20:37 smithi179 bash[74732]: ceph-44f886a8-dda7-11ee-95c6-87774f69a715-mds-l 2024-03-09T00:20:37.754 DEBUG:teuthology.orchestra.run.smithi179:> sudo pkill -f 'journalctl -f -n 0 -u ceph-44f886a8-dda7-11ee-95c6-87774f69a715@mds.l.service' 2024-03-09T00:20:37.788 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-09T00:20:37.788 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-09T00:20:37.789 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-09T00:20:37.979 INFO:teuthology.orchestra.run.smithi073.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:20:40.540 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-09T00:20:40.725 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:20:43.279 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force --keep-logs 2024-03-09T00:20:43.469 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:20:45.912 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:20:45.948 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:20:45.982 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-09T00:20:46.011 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-09T00:20:46.013 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi073/crash 2024-03-09T00:20:46.014 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash -- . 2024-03-09T00:20:46.228 INFO:teuthology.orchestra.run.smithi073.stderr:tar: /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-09T00:20:46.229 INFO:teuthology.orchestra.run.smithi073.stderr:tar: Error is not recoverable: exiting now 2024-03-09T00:20:46.231 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi102/crash 2024-03-09T00:20:46.232 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash -- . 2024-03-09T00:20:46.272 INFO:teuthology.orchestra.run.smithi102.stderr:tar: /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-09T00:20:46.273 INFO:teuthology.orchestra.run.smithi102.stderr:tar: Error is not recoverable: exiting now 2024-03-09T00:20:46.275 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi179/crash 2024-03-09T00:20:46.276 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash -- . 2024-03-09T00:20:46.315 INFO:teuthology.orchestra.run.smithi179.stderr:tar: /var/lib/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/crash: Cannot open: No such file or directory 2024-03-09T00:20:46.315 INFO:teuthology.orchestra.run.smithi179.stderr:tar: Error is not recoverable: exiting now 2024-03-09T00:20:46.327 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-09T00:20:46.328 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-09T00:20:46.370 INFO:tasks.cephadm:Compressing logs... 2024-03-09T00:20:46.370 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-09T00:20:46.412 DEBUG:teuthology.orchestra.run.smithi102:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-09T00:20:46.415 DEBUG:teuthology.orchestra.run.smithi179:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-09T00:20:46.439 INFO:teuthology.orchestra.run.smithi073.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-09T00:20:46.440 INFO:teuthology.orchestra.run.smithi073.stderr:: No such file or directory 2024-03-09T00:20:46.440 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.6.log 2024-03-09T00:20:46.440 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.a.log 2024-03-09T00:20:46.440 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-09T00:20:46.446 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.6.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.g.log 2024-03-09T00:20:46.446 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.j.log 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.3.log 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.x.log 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.3.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.0.log 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.x.log: 90.0% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr: 94.6% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.g.log.gz 2024-03-09T00:20:46.447 INFO:teuthology.orchestra.run.smithi073.stderr: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.j.log.gz 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.0.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log: 93.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr: 83.9% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.9.log 2024-03-09T00:20:46.460 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.d.log 2024-03-09T00:20:46.461 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.9.log: 87.3% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-09T00:20:46.461 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.a.log 2024-03-09T00:20:46.461 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.d.log: 94.6% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.d.log.gz 2024-03-09T00:20:46.461 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/ceph-client.admin.log 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:: No such file or directory 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.l.log 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.i.log 2024-03-09T00:20:46.465 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.l.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.11.log 2024-03-09T00:20:46.466 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.z.log 2024-03-09T00:20:46.466 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.i.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.8.log 2024-03-09T00:20:46.466 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-09T00:20:46.466 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.5.log 2024-03-09T00:20:46.466 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.8.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.f.log 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.5.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.f.log: 90.2% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log: 83.6% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.l.log.gz 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.2.log 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.c.log 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.2.log: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.f.log.gz 2024-03-09T00:20:46.467 INFO:teuthology.orchestra.run.smithi179.stderr: 87.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-09T00:20:46.468 INFO:teuthology.orchestra.run.smithi179.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-09T00:20:46.468 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.c.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.c.log 2024-03-09T00:20:46.468 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log: 92.0% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.z.log.gz 2024-03-09T00:20:46.468 INFO:teuthology.orchestra.run.smithi179.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.c.log: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.c.log.gz 2024-03-09T00:20:46.468 INFO:teuthology.orchestra.run.smithi179.stderr: 90.9% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-09T00:20:46.471 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.a.log: 94.6% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.a.log.gz 2024-03-09T00:20:46.471 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph-client.admin.log: 91.0% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-09T00:20:46.471 INFO:teuthology.orchestra.run.smithi073.stderr: 92.5% -- replaced with /var/log/ceph/ceph-client.admin.log.gz 2024-03-09T00:20:46.473 INFO:teuthology.orchestra.run.smithi102.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:: No such file or directory 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.10.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.b.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.10.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.4.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.1.log 2024-03-09T00:20:46.474 INFO:teuthology.orchestra.run.smithi102.stderr: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.k.log 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.h.log 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.k.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.7.log 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.e.log 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.h.log: /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.7.log: 90.1% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.audit.log.gz 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.y.log 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.e.log: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.k.log.gz 2024-03-09T00:20:46.475 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.y.log: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.h.log.gz 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr: 94.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.e.log.gz 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log: 83.7% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.cephadm.log.gz 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:gzip -5 --verbose -- /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.b.log 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log: 87.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph.log.gz 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr:/var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.b.log: 91.9% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.y.log.gz 2024-03-09T00:20:46.476 INFO:teuthology.orchestra.run.smithi102.stderr: 91.0% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-volume.log.gz 2024-03-09T00:20:46.728 INFO:teuthology.orchestra.run.smithi102.stderr: 91.9% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.b.log.gz 2024-03-09T00:20:46.787 INFO:teuthology.orchestra.run.smithi073.stderr: 89.8% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mgr.x.log.gz 2024-03-09T00:20:46.863 INFO:teuthology.orchestra.run.smithi179.stderr: 91.9% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.c.log.gz 2024-03-09T00:20:47.550 INFO:teuthology.orchestra.run.smithi179.stderr: 95.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.i.log.gz 2024-03-09T00:20:47.707 INFO:teuthology.orchestra.run.smithi073.stderr: 90.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mon.a.log.gz 2024-03-09T00:20:50.068 INFO:teuthology.orchestra.run.smithi102.stderr: 92.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.10.log.gz 2024-03-09T00:20:50.472 INFO:teuthology.orchestra.run.smithi073.stderr: 92.2% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.0.log.gz 2024-03-09T00:20:50.565 INFO:teuthology.orchestra.run.smithi102.stderr: 92.5% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.1.log.gz 2024-03-09T00:20:50.622 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.8.log.gz 2024-03-09T00:20:51.172 INFO:teuthology.orchestra.run.smithi179.stderr: 92.7% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.11.log.gz 2024-03-09T00:20:51.256 INFO:teuthology.orchestra.run.smithi073.stderr: 92.1% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.3.log.gz 2024-03-09T00:20:51.569 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.2.log.gz 2024-03-09T00:20:51.862 INFO:teuthology.orchestra.run.smithi073.stderr: 92.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.6.log.gz 2024-03-09T00:20:52.276 INFO:teuthology.orchestra.run.smithi102.stderr: 92.3% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.7.log.gz 2024-03-09T00:20:52.416 INFO:teuthology.orchestra.run.smithi073.stderr: 92.3% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.9.log.gz 2024-03-09T00:20:52.418 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-03-09T00:20:52.418 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m5.992s 2024-03-09T00:20:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m21.170s 2024-03-09T00:20:52.419 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.715s 2024-03-09T00:20:53.504 INFO:teuthology.orchestra.run.smithi179.stderr: 92.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.5.log.gz 2024-03-09T00:20:53.506 INFO:teuthology.orchestra.run.smithi179.stderr: 2024-03-09T00:20:53.506 INFO:teuthology.orchestra.run.smithi179.stderr:real 0m7.077s 2024-03-09T00:20:53.506 INFO:teuthology.orchestra.run.smithi179.stderr:user 0m21.925s 2024-03-09T00:20:53.506 INFO:teuthology.orchestra.run.smithi179.stderr:sys 0m0.719s 2024-03-09T00:20:54.753 INFO:teuthology.orchestra.run.smithi102.stderr: 92.8% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-osd.4.log.gz 2024-03-09T00:21:01.673 INFO:teuthology.orchestra.run.smithi102.stderr: 93.4% -- replaced with /var/log/ceph/44f886a8-dda7-11ee-95c6-87774f69a715/ceph-mds.b.log.gz 2024-03-09T00:21:01.675 INFO:teuthology.orchestra.run.smithi102.stderr: 2024-03-09T00:21:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:real 0m15.246s 2024-03-09T00:21:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:user 0m36.268s 2024-03-09T00:21:01.675 INFO:teuthology.orchestra.run.smithi102.stderr:sys 0m1.121s 2024-03-09T00:21:01.675 INFO:tasks.cephadm:Archiving logs... 2024-03-09T00:21:01.676 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi073/log 2024-03-09T00:21:01.677 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-09T00:21:05.099 DEBUG:teuthology.misc:Transferring archived files from smithi102:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi102/log 2024-03-09T00:21:05.100 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-09T00:21:10.686 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi179/log 2024-03-09T00:21:10.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-09T00:21:14.134 INFO:tasks.cephadm:Removing cluster... 2024-03-09T00:21:14.135 DEBUG:teuthology.orchestra.run.smithi073:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force 2024-03-09T00:21:14.328 INFO:teuthology.orchestra.run.smithi073.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:21:14.610 DEBUG:teuthology.orchestra.run.smithi102:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force 2024-03-09T00:21:14.801 INFO:teuthology.orchestra.run.smithi102.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:21:15.097 DEBUG:teuthology.orchestra.run.smithi179:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 44f886a8-dda7-11ee-95c6-87774f69a715 --force 2024-03-09T00:21:15.290 INFO:teuthology.orchestra.run.smithi179.stdout:Deleting cluster with fsid: 44f886a8-dda7-11ee-95c6-87774f69a715 2024-03-09T00:21:15.605 INFO:tasks.cephadm:Removing cephadm ... 2024-03-09T00:21:15.605 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-09T00:21:15.620 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-09T00:21:15.637 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-09T00:21:15.653 INFO:tasks.cephadm:Teardown complete 2024-03-09T00:21:15.654 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-09T00:21:15.667 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-09T00:21:15.668 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T00:21:15.670 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T00:21:15.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-09T00:21:15.799 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-09T00:21:15.810 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-09T00:21:15.810 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T00:21:15.813 DEBUG:teuthology.orchestra.run.smithi102:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T00:21:15.816 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-09T00:21:15.826 INFO:teuthology.orchestra.run.smithi073.stderr:bash: line 1: ntpq: command not found 2024-03-09T00:21:15.828 INFO:teuthology.orchestra.run.smithi102.stderr:bash: line 1: ntpq: command not found 2024-03-09T00:21:15.830 INFO:teuthology.orchestra.run.smithi179.stderr:bash: line 1: ntpq: command not found 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:=============================================================================== 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 26 +176us[ +151us] +/- 137ms 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 25 -248us[ -273us] +/- 62ms 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:^* hv03.front.sepia.ceph.com 3 6 377 24 +206us[ +181us] +/- 86ms 2024-03-09T00:21:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T00:21:15.832 INFO:teuthology.orchestra.run.smithi102.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T00:21:15.833 INFO:teuthology.orchestra.run.smithi102.stdout:=============================================================================== 2024-03-09T00:21:15.833 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 28 +137us[ +137us] +/- 136ms 2024-03-09T00:21:15.833 INFO:teuthology.orchestra.run.smithi102.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 27 -276us[ -276us] +/- 62ms 2024-03-09T00:21:15.833 INFO:teuthology.orchestra.run.smithi102.stdout:^* hv03.front.sepia.ceph.com 3 6 377 30 +176us[ +189us] +/- 86ms 2024-03-09T00:21:15.833 INFO:teuthology.orchestra.run.smithi102.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T00:21:15.836 INFO:teuthology.orchestra.run.smithi179.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-09T00:21:15.837 INFO:teuthology.orchestra.run.smithi179.stdout:=============================================================================== 2024-03-09T00:21:15.837 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 30 +165us[ +152us] +/- 136ms 2024-03-09T00:21:15.837 INFO:teuthology.orchestra.run.smithi179.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 29 -249us[ -263us] +/- 62ms 2024-03-09T00:21:15.837 INFO:teuthology.orchestra.run.smithi179.stdout:^* hv03.front.sepia.ceph.com 3 7 377 27 +229us[ +215us] +/- 86ms 2024-03-09T00:21:15.837 INFO:teuthology.orchestra.run.smithi179.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-09T00:21:15.838 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-09T00:21:15.849 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-09T00:21:15.850 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-09T00:21:15.862 DEBUG:teuthology.orchestra.run.smithi073:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-09T00:21:15.953 DEBUG:teuthology.orchestra.run.smithi102:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-09T00:21:16.026 DEBUG:teuthology.orchestra.run.smithi179:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-09T00:21:16.097 DEBUG:teuthology.orchestra.run.smithi073:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-09T00:21:16.128 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:16.128 DEBUG:teuthology.orchestra.run.smithi102:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-09T00:21:16.158 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:16.159 DEBUG:teuthology.orchestra.run.smithi179:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-09T00:21:16.188 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:16.189 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-09T00:21:16.202 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-09T00:21:16.233 INFO:teuthology.task.internal:Duration was 1990.365054 seconds 2024-03-09T00:21:16.234 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-09T00:21:16.289 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-09T00:21:16.290 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T00:21:16.293 DEBUG:teuthology.orchestra.run.smithi102:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T00:21:16.298 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-09T00:21:16.334 INFO:teuthology.orchestra.run.smithi073.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T00:21:16.338 INFO:teuthology.orchestra.run.smithi102.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T00:21:16.342 INFO:teuthology.orchestra.run.smithi179.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-09T00:21:16.794 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-09T00:21:16.794 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2024-03-09T00:21:16.795 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T00:21:16.816 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi102.front.sepia.ceph.com 2024-03-09T00:21:16.816 DEBUG:teuthology.orchestra.run.smithi102:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T00:21:16.853 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2024-03-09T00:21:16.854 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-09T00:21:16.873 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-09T00:21:16.874 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T00:21:16.876 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T00:21:16.897 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-09T00:21:16.942 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-09T00:21:16.942 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T00:21:16.946 DEBUG:teuthology.orchestra.run.smithi102:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T00:21:16.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-09T00:21:17.336 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-09T00:21:17.351 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-09T00:21:17.351 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T00:21:17.380 DEBUG:teuthology.orchestra.run.smithi102:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T00:21:17.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-09T00:21:17.439 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-09T00:21:17.721 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.730 DEBUG:teuthology.orchestra.run.smithi102:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.734 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.751 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2024-03-09T00:21:17.758 INFO:teuthology.orchestra.run.smithi102.stdout:kernel.core_pattern = core 2024-03-09T00:21:17.761 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2024-03-09T00:21:17.801 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:17.828 DEBUG:teuthology.orchestra.run.smithi102:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.861 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:17.862 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-09T00:21:17.881 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-09T00:21:17.882 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-09T00:21:18.038 INFO:teuthology.task.internal:Transferring archived files... 2024-03-09T00:21:18.038 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi073 2024-03-09T00:21:18.039 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T00:21:18.201 DEBUG:teuthology.misc:Transferring archived files from smithi102:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi102 2024-03-09T00:21:18.202 DEBUG:teuthology.orchestra.run.smithi102:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T00:21:18.360 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328/remote/smithi179 2024-03-09T00:21:18.360 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-09T00:21:18.437 INFO:teuthology.task.internal:Removing archive directory... 2024-03-09T00:21:18.438 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T00:21:18.441 DEBUG:teuthology.orchestra.run.smithi102:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T00:21:18.444 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-09T00:21:18.463 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-09T00:21:18.492 INFO:teuthology.task.internal:Not uploading archives. 2024-03-09T00:21:18.492 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-09T00:21:18.520 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-09T00:21:18.520 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T00:21:18.524 DEBUG:teuthology.orchestra.run.smithi102:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T00:21:18.529 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-09T00:21:18.537 INFO:teuthology.orchestra.run.smithi073.stdout: 265938 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 00:21 /home/ubuntu/cephtest 2024-03-09T00:21:18.540 INFO:teuthology.orchestra.run.smithi102.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 00:21 /home/ubuntu/cephtest 2024-03-09T00:21:18.549 INFO:teuthology.orchestra.run.smithi179.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 9 00:21 /home/ubuntu/cephtest 2024-03-09T00:21:18.550 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-09T00:21:18.601 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-09T00:21:19.003 INFO:teuthology.nuke:Checking targets against current locks 2024-03-09T00:21:19.039 DEBUG:teuthology.nuke:shortname: smithi073 2024-03-09T00:21:19.039 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T00:21:19.088 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.163132', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJPscUV6Gm6IOLkVUrW7zI6I49cIDeUJYfmsfXdJ+pGNir7An9u17Nx42xmBpog0CGG3wCojiytGRM3ZJfqHE+A='} 2024-03-09T00:21:19.096 DEBUG:teuthology.nuke:shortname: smithi102 2024-03-09T00:21:19.096 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T00:21:19.138 DEBUG:teuthology.nuke:shortname: smithi179 2024-03-09T00:21:19.138 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-09T00:21:19.150 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi102.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.161609', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBK2GvsZBXQDLL/W3R/OhFX9IjxjwLYTlsB037IJvqxbqbKWiyFE+B4hCNfXcx1TRCjGJk3Gy+B1paKxL2hqjlF0='} 2024-03-09T00:21:19.194 INFO:teuthology.orchestra.console.smithi073:Power off 2024-03-09T00:21:19.194 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T00:21:19.212 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/rishabh-2024-03-08_19:54:47-fs-rishabh-2024mar8-testing-default-smithi/7588328', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-08 23:39:47.164399', 'locked_by': 'scheduled_rishabh@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBE9LxVk+vl16XT+ugVB8cplHsXPPuF9kA+ghz4vJ3M+6u8igdtPgHDPWuSv8zrS7ViQdzMzMzIcf0UTDE/QEw0o='} 2024-03-09T00:21:19.232 DEBUG:teuthology.orchestra.console.smithi073:power off output: Chassis Power Control: Down/Off 2024-03-09T00:21:19.232 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:19.262 INFO:teuthology.orchestra.console.smithi102:Power off 2024-03-09T00:21:19.263 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T00:21:19.298 DEBUG:teuthology.orchestra.console.smithi102:power off output: Chassis Power Control: Down/Off 2024-03-09T00:21:19.298 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:19.314 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-03-09T00:21:19.323 INFO:teuthology.orchestra.console.smithi179:Power off 2024-03-09T00:21:19.323 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-09T00:21:19.339 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-09T00:21:19.358 DEBUG:teuthology.orchestra.console.smithi179:power off output: Chassis Power Control: Down/Off 2024-03-09T00:21:19.358 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:19.393 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-09T00:21:23.316 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:23.340 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:23.395 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:23.430 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is on 2024-03-09T00:21:23.457 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is on 2024-03-09T00:21:23.507 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is on 2024-03-09T00:21:27.434 DEBUG:teuthology.orchestra.console.smithi073:pexpect command: ipmitool -H smithi073.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:27.458 DEBUG:teuthology.orchestra.console.smithi102:pexpect command: ipmitool -H smithi102.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:27.508 DEBUG:teuthology.orchestra.console.smithi179:pexpect command: ipmitool -H smithi179.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-09T00:21:27.552 DEBUG:teuthology.orchestra.console.smithi073:check power output: Chassis Power is off 2024-03-09T00:21:27.570 DEBUG:teuthology.orchestra.console.smithi102:check power output: Chassis Power is off 2024-03-09T00:21:27.622 DEBUG:teuthology.orchestra.console.smithi179:check power output: Chassis Power is off 2024-03-09T00:21:27.653 INFO:teuthology.orchestra.console.smithi073:Power off completed 2024-03-09T00:21:27.671 INFO:teuthology.orchestra.console.smithi102:Power off completed 2024-03-09T00:21:27.723 INFO:teuthology.orchestra.console.smithi179:Power off completed 2024-03-09T00:21:27.830 INFO:teuthology.run:Summary data: description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v1} mount overrides/{distro/stock/{centos_9.stream k-stock} ms-die-on-skipped}} ms_mode/legacy wsync/no} objectstore-ec/bluestore-ec-root omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/1 standby-replay tasks/{0-subvolume/{with-namespace-isolated-and-quota} 1-check-counter 2-scrub/yes 3-snaps/yes 4-flush/yes 5-workunit/suites/fsx}} duration: 1990.3650543689728 failure_reason: 'Command failed (workunit test suites/fsx.sh) on smithi073 with status 2: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=7d63482aa89628d733e298984a762389b625c761 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsx.sh''' flavor: default owner: scheduled_rishabh@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=bc7102cb35984a438080376b4db19bbe status: fail success: false 2024-03-09T00:21:27.830 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-09T00:21:27.926 INFO:teuthology.run:FAIL